Voice cloning method and device based on emotion enhancement and related medium

Abstract

The invention discloses a speech cloning method and device based on emotion enhancement, and a related medium. The method comprises the following steps: respectively obtaining a reference audio and a prediction text corresponding to the reference audio; respectively preprocessing the reference audio and the prediction text to generate standardized audio data for feature extraction; inputting the standardized audio data into an emotion enhancement module so as to generate acoustic features matched with a target voice style through multi-round feature fusion and an autoregression generation mechanism; and inputting the acoustic features into a speech synthesis module for decoding processing, and outputting predicted speech. According to the invention, by introducing the emotion enhancement module, accurate modeling of the emotion style of the speaker in the acoustic feature generation process is realized, so that the emotion simulation degree and the personality restoration capability of the synthesized speech are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of multimedia processing technology, and in particular to a voice cloning method and device based on emotion enhancement and related media. Background Art

[0002] Existing voice cloning methods generally have several technical bottlenecks. On the one hand, current mainstream methods rely on large amounts of high-quality, consistently paired voice data for training in order to learn sufficiently rich speaker features. When the scale of training data is insufficient, or the data fails to cover specific timbres, intonations, and emotional expression styles, the quality of the synthesized speech is prone to a significant decline, manifesting as inconsistent timbre, rhythmic deviations, and lack of emotion. On the other hand, when faced with zero-shot or few-shot speech synthesis tasks, traditional models find it difficult to fully capture the fine-grained emotional signals in the reference audio, resulting in a lack of natural, personalized expression in the output speech. In addition, existing methods are relatively weak in emotion modeling and often fail to effectively integrate emotional and semantic features, making it difficult to achieve accurate audio synthesis based on the target text content and the emotional style of the reference audio. Summary of the Invention

[0003] The embodiments of the present invention provide a voice cloning method, device and related medium based on emotion enhancement, aiming to solve the problem in the prior art that voice cloning has low timbre and emotion restoration, resulting in unnatural emotional expression.

[0004] In a first aspect, an embodiment of the present invention provides a voice cloning method based on emotion enhancement, comprising:

[0005] Respectively obtaining a reference audio and a predicted text corresponding to the reference audio;

[0006] Preprocessing the reference audio and predicted text respectively to generate standardized audio data for feature extraction;

[0007] Inputting the standardized audio data into an emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism;

[0008] The acoustic features are input into the speech synthesis module for decoding and output as predicted speech.

[0009] In a second aspect, an embodiment of the present invention provides a voice cloning device based on emotion enhancement, comprising:

[0010] an audio extraction unit, configured to respectively obtain a reference audio and a predicted text corresponding to the reference audio;

[0011] an audio processing unit, configured to pre-process the reference audio and the predicted text respectively to generate standardized audio data for feature extraction;

[0012] A data fusion unit, configured to input the standardized audio data into an emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism;

[0013] The speech output unit is used to input the acoustic features into the speech synthesis module for decoding processing and output the predicted speech.

[0014] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the emotion-enhanced voice cloning method of the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the emotion-enhanced-based voice cloning method of the first aspect is implemented.

[0016] An embodiment of the present invention provides a speech cloning method based on emotion enhancement, comprising obtaining reference audio and predicted text corresponding to the reference audio; preprocessing the reference audio and predicted text to generate standardized audio data for feature extraction; inputting the standardized audio data into an emotion enhancement module to generate acoustic features matching the target speech style through multiple rounds of feature fusion and autoregressive generation; and inputting the acoustic features into a speech synthesis module for decoding and outputting predicted speech. By introducing the emotion enhancement module, the present invention achieves accurate modeling of the speaker's emotional style during the acoustic feature generation process, thereby significantly improving the emotional fidelity and individuality of the synthesized speech.

[0017] The embodiment of the present invention also provides a voice cloning device, a computer device and a storage medium based on emotion enhancement, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1A flowchart of a voice cloning method based on emotion enhancement provided by an embodiment of the present invention;

[0020] Figure 2 Another schematic flow chart of a voice cloning method based on emotion enhancement provided by an embodiment of the present invention;

[0021] Figure 3 A schematic block diagram of a voice cloning device based on emotion enhancement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] See below Figure 1 , Figure 1 A flowchart of a voice cloning method based on emotion enhancement provided by an embodiment of the present invention specifically includes steps S101 to S104.

[0027] S101, respectively obtaining a reference audio and a predicted text corresponding to the reference audio;

[0028] S102, preprocessing the reference audio and predicted text respectively to generate standardized audio data for feature extraction;

[0029] S103: Inputting the standardized audio data into an emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism;

[0030] S104: Input the acoustic features into a speech synthesis module for decoding and outputting predicted speech.

[0031] Combine Figure 2 As shown, in step S101, the reference audio may be any 3 to 10 seconds of speech data provided by the target speaker, and the predicted text is the text content required to be read aloud for the speech to be synthesized.

[0032] In step S102, data preprocessing is performed on the reference audio and the predicted text respectively to obtain standardized audio data in a unified format.

[0033] In one embodiment, step S102 includes:

[0034] Inputting the reference audio into an acoustic feature extraction model to extract an acoustic feature vector corresponding to the reference audio;

[0035] Inputting the reference audio into an emotion recognition model to extract an emotion feature vector corresponding to the reference audio;

[0036] Inputting the predicted text into a text feature extraction model to extract a semantic feature vector corresponding to the predicted text;

[0037] The acoustic feature vector, the emotional feature vector and the semantic feature vector are respectively integrated to obtain standardized audio data.

[0038] In this embodiment, the reference audio is input into the HuBERT pre-trained model for processing, and an acoustic feature vector containing global speech information can be extracted. This acoustic feature vector can effectively describe the speaker's voice attributes such as timbre and prosody, providing basic support for voice cloning. The Wav2Vec2.0 SER model can be used as an emotion recognition model to extract the emotion feature vector of the reference audio. For the extraction of predictive text, the BERT model can be used to model the input text and obtain a semantic feature vector.

[0039] Finally, the three types of features described above—the acoustic feature vectors, the emotional feature vectors, and the semantic feature vectors—are integrated and processed according to the requirement for uniform feature dimensionality. Integration methods may include vector-level concatenation, linear mapping, or normalization to generate standardized audio data in a unified format, which serves as input for the subsequent emotion enhancement module. Through these steps, the combined expression of multimodal features can be effectively achieved, ensuring that emotional, speech, and semantic information are fully preserved and utilized during speech generation.

[0040] In one embodiment, inputting the reference audio into an acoustic feature extraction model to extract an acoustic feature vector corresponding to the reference audio includes:

[0041] Setting the reference audio to a plurality of sampling points arranged in time sequence;

[0042] Inputting the sampling points into a multi-layer convolutional encoder for downsampling to obtain intermediate acoustic features;

[0043] The intermediate acoustic features are input into an encoder based on the attention mechanism to extract global contextual information and generate an acoustic feature vector.

[0044] In this embodiment, the reference audio is set to a plurality of sampling points arranged in time sequence, denoted as X={x1, x2, ..., x T}, where T represents the total number of sampling points of the reference audio, x t Represents the audio sampling signal at the t-th time point. This sequence is used as the model input for the subsequent hierarchical extraction of acoustic features. The above sampling points are input to the multi-layer convolutional encoder (ConvEncoder) for downsampling processing to extract intermediate acoustic features. The convolutional encoder extracts the short-term feature information of the audio in the local time window through several convolution kernels, and realizes downsampling on the time axis through the step operation, and outputs the intermediate acoustic features Z = {z1,z2,...,z T'}, where T'<T, represents the number of time steps after downsampling.

[0045] Furthermore, the intermediate acoustic features Z are fed into an attention-based Transformer encoder to further extract the global contextual information of the reference audio. The Transformer encoder calculates the relative importance of different time steps through a multi-head self-attention mechanism and performs a weighted combination of the features of each time step to generate a context-aware acoustic feature vector H = Transformer(Z). This acoustic feature vector serves as the acoustic representation of the reference audio for subsequent speech generation tasks.

[0046] Since the HuBERT pre-trained model is obtained through self-supervised learning training based on large-scale unlabeled speech data, it has stronger modeling and feature generalization capabilities than traditional acoustic feature extraction technologies. Therefore, HuBERT is used as the acoustic feature extraction model in the present invention, which can more effectively represent the timbre, rhythm and other speech characteristics in the reference audio.

[0047] In one embodiment, inputting the reference audio into an emotion recognition model to extract an emotion feature vector corresponding to the reference audio includes:

[0048] Summing the acoustic eigenvectors and dividing the sum by the total number of vectors to obtain an average eigenvector;

[0049] Inputting the average feature vector into a classifier for normalization calculation, and outputting the probability distribution of the reference audio under various emotion labels;

[0050] Based on the cross entropy loss between the probability distribution and the target emotion label, the classification ability of the emotion recognition model is optimized, and an emotion feature vector is output.

[0051] In this embodiment, the reference audio is used as input data and fed into the Wav2Vec2.0 SER model for processing. Downsampling is performed through a multi-layer convolutional encoder, and the acoustic feature vector C of the reference audio is output using a Transformer encoder. T' The model includes a pre-trained Wav2Vec2.0 model and an emotion recognition module fine-tuned for the Speech Emotion Recognition (SER) task. The acoustic feature vectors are average pooled over time to generate an average feature vector h of fixed dimension. The specific process is as follows: the feature vectors c of all acoustic feature vector time steps are pooled. t After summing, divide by the number of time steps T'. The specific formula is as follows:

[0052]

[0053] The average feature vector h is used as the feature input of the emotion recognition model to represent the emotional information of the entire reference audio. The emotional feature vector h is then input into the softmax classifier for normalization to output the probability distribution of all emotional labels for the reference audio. The specific formula is as follows:

[0054]

[0055] Among them, W and b are the weight matrix and bias term obtained from model training, Represents the probability distribution of the emotion category prediction corresponding to the reference audio.

[0056] During the training phase, the sentiment prediction probability distribution of the model output is further calculated The cross entropy loss function L between the true emotional label y SER , which is used to optimize the classification performance of the emotion recognition model. The specific formula is as follows:

[0057]

[0058] The cross-entropy loss function measures the deviation between the model's predictions and the actual emotion labels, guiding the updating and optimization of model parameters. During the inference phase, there's no need to calculate the loss function; instead, the average pooled emotion feature vector is retained and serves as the key input for the emotion enhancement module in the subsequent voice cloning process.

[0059] In one embodiment, inputting the predicted text into a text feature extraction model to extract a semantic feature vector corresponding to the predicted text includes:

[0060] Perform word segmentation on the predicted text to obtain a text sequence, and convert the text sequence into a corresponding word embedding vector;

[0061] Based on the position information of each word in the text sequence and the segment information of the sentence in which each word is located, respectively generating a position embedding vector and a segment embedding vector;

[0062] The word embedding vector, position embedding vector and segment embedding vector are added and fused respectively to obtain a semantic feature vector.

[0063] In this embodiment, the BERT model can be used as a text feature extraction model, and its word segmentation strategy adopts the WordPiece word segmentation algorithm. Through this word segmentation mechanism, the original text can be split into smaller subwords or word fragments, thereby improving the coverage and flexibility of word embedding representation, and finally obtaining a set of ordered text sequences. The text sequence is converted into the corresponding word embedding vector E token At the same time, in order to improve the model's ability to perceive text structure, it is also necessary to introduce the position embedding vector E position and the segment embedding vector E segment , which are used to represent the absolute position of each word in the text and the sentence paragraph to which it belongs. The three vectors can together form the final fusion vector E, and the specific formula is as follows:

[0064] E=E token +E segment +E position

[0065] The fused vector E is input into the Transformer encoder for feature modeling. The Transformer encoder consists of several encoding layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network module. Through the self-attention mechanism, the model can dynamically capture the dependencies and semantic connections between words in the input sequence; and under the joint action of the multi-layer structure, it extracts multi-granular contextual information and enhances the global semantic associations between words, thereby obtaining a more accurate and abstract text semantic representation. Finally, through the processing of the above-mentioned Transformer encoder, a semantic feature vector corresponding to the semantic content of the input text is output. As an important component of standardized audio data, this semantic feature vector provides accurate language-level expression support for the subsequent speech generation process.

[0066] In one embodiment, the word embedding vector, the position embedding vector, and the segment embedding vector are added and fused to obtain a semantic feature vector, including:

[0067] Inputting the semantic feature vector into an encoder based on an attention mechanism to utilize the attention mechanism to obtain the contextual dependencies between words;

[0068] Based on the context dependency, local semantics and global semantics are captured respectively through a multi-layer structure, and a semantic feature vector representing the semantics of the predicted text is output.

[0069] In this embodiment, the semantic feature vector is input into an encoder based on an attention mechanism for deep semantic modeling. In this embodiment, a multi-layer Transformer encoder structure is used, the core of which is a combination of a multi-head self-attention mechanism and a feedforward neural network module. The multi-head attention mechanism can calculate the attention weight of each word in parallel with its attention weights for other words, thereby mining the contextual dependencies between words and realizing dynamic modeling of semantic connections.

[0070] Furthermore, by stacking multiple Transformer encoding layers, semantic features are abstractly expressed at different levels. Lower layers primarily extract local semantic information, such as phrase collocation and grammatical structure, while higher layers focus on global semantic modeling, such as logical relationships between sentences and contextual consistency. Through multi-level feature transfer and fusion, the final output is a structured, context-sensitive, and semantically complete semantic feature vector, which is used to guide the precise alignment of speech style and semantic content in subsequent speech synthesis.

[0071] In step S103, the emotion enhancement module can generate a structural design based on the SE-GPT model, and construct a progressive acoustic anchor point sequence through an autoregressive generation mechanism. In each step, it relies on the anchor point generated at the previous moment, which can achieve modeling of context coherence and rhythm characteristics, and finally output complete acoustic features.

[0072] Specifically, the feature input method of the SE-GPT model is different from that of the traditional GPT model. The complete workflow is as follows: first, the emotional features of the reference audio are linearly mapped and normalized, and their dimensions are adjusted to be consistent with the text semantic features. Then, the emotional features and the corresponding text features are summed at the vector level to form the fused emotional semantic features. At the same time, the acoustic feature anchors generated by the reference audio after HuBERT encoding will be embedded (Embedding) as an input vector sequence, and its vector dimension is consistent with the fused feature. Finally, the emotional semantic features and the acoustic anchor sequence are merged to form the complete model input.

[0073] The merged feature sequence is fed into the Transformer decoder in the SE-GPT module for layer-by-layer processing. This decoder uses a multi-head self-attention mechanism stacked on top of multiple layers to contextually model the input features, capturing the long-term dependencies between semantics and sentiment, thereby more accurately predicting each acoustic anchor point. The SE-GPT module employs an autoregressive generation mechanism, outputting only one acoustic anchor point at a time and using the current generation result as part of the input for the next moment, enabling sequence-level progressive prediction. If the output anchor point is a termination marker, the acoustic anchor point sequence generation is complete; if not, iteration continues until the complete acoustic feature is generated.

[0074] In one embodiment, step S103 includes:

[0075] Inputting the standardized audio data as a fusion input guide item into the emotion enhancement module; wherein the standardized audio data includes a semantic feature vector, an emotional feature vector and an acoustic feature vector;

[0076] The semantic feature vector and the emotional feature vector are coupled and fused respectively to generate a fused emotional semantic vector;

[0077] The fused emotional semantic vector and the acoustic feature vector are subjected to multiple rounds of interactive fusion processing, and an autoregressive generation path is constructed. The output of the previous moment is used as the input reference of the current moment in turn, and finally the acoustic features generated by fusion and autoregression are output.

[0078] In this embodiment, the standardized audio data is input into the emotion enhancement module as a fusion input guide item. The standardized audio data includes the semantic feature vector, emotional feature vector and acoustic feature vector extracted by the previous step, which are respectively used to characterize the predicted text content, the emotional state of the reference audio, and the timbre and rhythmic attributes of the speaker. The semantic feature vector and the emotional feature vector are then coupled and fused at the feature level. Specifically, after the emotional feature vector is linearly mapped and normalized, it is added to the semantic feature vector at the element level to generate a fused emotional semantic vector, thereby realizing unified modeling of language content and emotional expression in the vector space.

[0079] Furthermore, the fused emotional semantic vector and acoustic feature vector are input into the speech generation module based on the GPT structure for multiple rounds of interactive fusion. This module adopts a multi-layer Transformer decoder structure, and gradually screens and reconstructs the fused input through a multi-head attention mechanism, and introduces an autoregressive generation path to model the contextual dependency between time series. That is, at each moment, the model makes the next prediction based on the intermediate acoustic feature anchor generated at the previous moment, combined with the current input feature, to form a progressive generation process. Finally, the acoustic features processed by multiple rounds of feature fusion and autoregressive mechanism are output. These acoustic features have semantic consistency, emotional expression integrity, and timbre style continuity, and serve as the input for decoding in the subsequent speech synthesis module, providing structural feature support for generating high-quality, style-matched synthetic speech.

[0080] In step S104, the SoVITS speech synthesis framework can be used to combine the acoustic features with the existing voice codebook mapping relationship to decode the acoustic features into a high-quality, natural predicted speech output with timbre and emotion consistent with the reference audio.

[0081] In summary, the present invention provides a speech cloning method based on emotion enhancement. First, a pre-trained model is used to extract acoustic, emotional, and semantic feature vectors, respectively, and these three feature vectors are fused to form standardized audio data. Subsequently, based on the SE-GPT generation module, multiple rounds of feature interaction and autoregressive generation paths are constructed to effectively output acoustic features. Finally, high-quality, stylistically consistent synthesized speech is generated through decoding via the SoVITS speech synthesis module. In this way, the present invention can enhance the consistency and naturalness of emotional expression during speech synthesis. The emotion recognition module extracts emotional features from the reference audio and deeply integrates them with the text semantics. This ensures that the resulting speech is not only accurate in content but also truly expresses the corresponding emotional state, enhancing the voice's appeal and anthropomorphism. Furthermore, it helps improve the accuracy of timbre restoration. By extracting detailed acoustic features using the HuBERT model and combining it with the contextual modeling capabilities of the SE-GPT model in autoregressive generation, the generated speech is closer to the original reference audio in terms of timbre and rhythm, enhancing the personalized expressiveness of the speech.

[0082] In addition, the present invention can achieve semantic-driven adaptive speech generation. Through the BERT model, the predicted text is understood in context, so that the speech synthesis system is consistent with the text semantics in terms of stress, intonation, speaking speed, etc., thereby improving the naturalness and semantic integrity of the speech. The present invention supports speech cloning under zero-sample or small-sample conditions, effectively overcoming the problem of traditional methods' strong dependence on large-scale speech corpus, and is suitable for multi-language, multi-scenario, and multi-style speech generation needs. The present invention has significant technical advantages in improving the naturalness, personalization, and emotional expression capabilities of the speech cloning system, and can be widely used in multiple fields such as virtual digital humans, voice assistants, and audio content creation, with good application prospects and commercial value.

[0083] Combine Figure 3 As shown, Figure 3 A schematic block diagram of a speech cloning device based on emotion enhancement provided by an embodiment of the present invention is provided. The speech cloning device based on emotion enhancement 300 includes:

[0084] The audio extraction unit 301 is configured to obtain a reference audio and a predicted text corresponding to the reference audio;

[0085] An audio processing unit 302, configured to pre-process the reference audio and the predicted text respectively to generate standardized audio data for feature extraction;

[0086] The data fusion unit 303 is used to input the standardized audio data into the emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism;

[0087] The speech output unit 304 is configured to input the acoustic features into the speech synthesis module for decoding and output the predicted speech.

[0088] In this embodiment, the audio extraction unit 301 obtains reference audio and predicted text corresponding to the reference audio respectively; the audio processing unit 302 preprocesses the reference audio and predicted text respectively to generate standardized audio data for feature extraction; the data fusion unit 303 inputs the standardized audio data into the emotion enhancement module to generate acoustic features matching the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism; the speech output unit 304 inputs the acoustic features into the speech synthesis module for decoding processing and outputs the predicted speech.

[0089] In one embodiment, the audio processing unit 302 is specifically configured to:

[0090] Inputting the reference audio into an acoustic feature extraction model to extract an acoustic feature vector corresponding to the reference audio;

[0091] Inputting the reference audio into an emotion recognition model to extract an emotion feature vector corresponding to the reference audio;

[0092] Inputting the predicted text into a text feature extraction model to extract a semantic feature vector corresponding to the predicted text;

[0093] The acoustic feature vector, the emotional feature vector and the semantic feature vector are respectively integrated to obtain standardized audio data.

[0094] In one embodiment, the audio processing unit 302 is further configured to:

[0095] Setting the reference audio to a plurality of sampling points arranged in time sequence;

[0096] Inputting the sampling points into a multi-layer convolutional encoder for downsampling to obtain intermediate acoustic features;

[0097] The intermediate acoustic features are input into an encoder based on the attention mechanism to extract global contextual information and generate an acoustic feature vector.

[0098] In one embodiment, the audio processing unit 302 is further configured to:

[0099] Summing the acoustic eigenvectors and dividing the sum by the total number of vectors to obtain an average eigenvector;

[0100] Inputting the average feature vector into a classifier for normalization calculation, and outputting the probability distribution of the reference audio under various emotion labels;

[0101] Based on the cross entropy loss between the probability distribution and the target emotion label, the classification ability of the emotion recognition model is optimized, and an emotion feature vector is output.

[0102] In one embodiment, the audio processing unit 302 is further configured to:

[0103] Perform word segmentation on the predicted text to obtain a text sequence, and convert the text sequence into a corresponding word embedding vector;

[0104] Based on the position information of each word in the text sequence and the segment information of the sentence in which each word is located, respectively generating a position embedding vector and a segment embedding vector;

[0105] The word embedding vector, position embedding vector and segment embedding vector are added and fused respectively to obtain a semantic feature vector.

[0106] In one embodiment, the audio processing unit 302 is further configured to:

[0107] Inputting the semantic feature vector into an encoder based on an attention mechanism to utilize the attention mechanism to obtain the contextual dependencies between words;

[0108] Based on the context dependency, local semantics and global semantics are captured respectively through a multi-layer structure, and a semantic feature vector representing the semantics of the predicted text is output.

[0109] In one embodiment, the data fusion unit 303 is specifically configured to:

[0110] Inputting the standardized audio data as a fusion input guide item into the emotion enhancement module; wherein the standardized audio data includes a semantic feature vector, an emotional feature vector and an acoustic feature vector;

[0111] The semantic feature vector and the emotional feature vector are coupled and fused respectively to generate a fused emotional semantic vector;

[0112] The fused emotional semantic vector and the acoustic feature vector are subjected to multiple rounds of interactive fusion processing, and an autoregressive generation path is constructed. The output of the previous moment is used as the input reference of the current moment in turn, and finally the acoustic features generated by fusion and autoregression are output.

[0113] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, and they will not be repeated here.

[0114] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, the steps provided in the above embodiments can be implemented. The storage medium may include a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0115] The present invention also provides a computer device that may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, the steps provided in the above embodiment can be implemented. Of course, the computer device may also include various network interfaces, a power supply, and other components.

[0116] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0117] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusions, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A voice cloning method based on emotion enhancement, characterized in that: include: Respectively obtaining a reference audio and a predicted text corresponding to the reference audio; Preprocessing the reference audio and predicted text respectively to generate standardized audio data for feature extraction; Inputting the standardized audio data into an emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism; The acoustic features are input into the speech synthesis module for decoding and output as predicted speech.

2. The voice cloning method based on emotion enhancement according to claim 1, characterized in that: The preprocessing of the reference audio and the predicted text respectively to generate standardized audio data for feature extraction includes: Inputting the reference audio into an acoustic feature extraction model to extract an acoustic feature vector corresponding to the reference audio; Inputting the reference audio into an emotion recognition model to extract an emotion feature vector corresponding to the reference audio; Inputting the predicted text into a text feature extraction model to extract a semantic feature vector corresponding to the predicted text; The acoustic feature vector, the emotional feature vector and the semantic feature vector are respectively integrated to obtain standardized audio data.

3. The voice cloning method based on emotion enhancement according to claim 2, characterized in that: Inputting the reference audio into an acoustic feature extraction model to extract an acoustic feature vector corresponding to the reference audio includes: Setting the reference audio to a plurality of sampling points arranged in time sequence; Inputting the sampling points into a multi-layer convolutional encoder for downsampling to obtain intermediate acoustic features; The intermediate acoustic features are input into an encoder based on the attention mechanism to extract global contextual information and generate an acoustic feature vector.

4. The voice cloning method based on emotion enhancement according to claim 3 is characterized in that: Inputting the reference audio into an emotion recognition model to extract an emotion feature vector corresponding to the reference audio includes: Summing the acoustic eigenvectors and dividing the sum by the total number of vectors to obtain an average eigenvector; Inputting the average feature vector into a classifier for normalization calculation, and outputting the probability distribution of the reference audio under various emotion labels; Based on the cross entropy loss between the probability distribution and the target emotion label, the classification ability of the emotion recognition model is optimized, and an emotion feature vector is output.

5. The voice cloning method based on emotion enhancement according to claim 2, characterized in that: Inputting the predicted text into a text feature extraction model to extract a semantic feature vector corresponding to the predicted text includes: Perform word segmentation on the predicted text to obtain a text sequence, and convert the text sequence into a corresponding word embedding vector; Based on the position information of each word in the text sequence and the segment information of the sentence in which each word is located, respectively generating a position embedding vector and a segment embedding vector; The word embedding vector, position embedding vector and segment embedding vector are added and fused respectively to obtain a semantic feature vector.

6. The voice cloning method based on emotion enhancement according to claim 5, characterized in that: The word embedding vector, position embedding vector and segment embedding vector are respectively added and fused to obtain a semantic feature vector, including: Inputting the semantic feature vector into an encoder based on an attention mechanism to utilize the attention mechanism to obtain the contextual dependencies between words; Based on the context dependency, local semantics and global semantics are captured respectively through a multi-layer structure, and a semantic feature vector representing the semantics of the predicted text is output.

7. The voice cloning method based on emotion enhancement according to claim 1, characterized in that: The standardized audio data is input into the emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism, including: Inputting the standardized audio data as a fusion input guide item into the emotion enhancement module; wherein the standardized audio data includes a semantic feature vector, an emotional feature vector and an acoustic feature vector; The semantic feature vector and the emotional feature vector are coupled and fused respectively to generate a fused emotional semantic vector; The fused emotional semantic vector and the acoustic feature vector are subjected to multiple rounds of interactive fusion processing, and an autoregressive generation path is constructed. The output of the previous moment is used as the input reference of the current moment in turn, and finally the acoustic features generated by fusion and autoregression are output.

8. A voice cloning device based on emotion enhancement, characterized in that: include: an audio extraction unit, configured to respectively obtain a reference audio and a predicted text corresponding to the reference audio; an audio processing unit, configured to pre-process the reference audio and the predicted text respectively to generate standardized audio data for feature extraction; A data fusion unit, configured to input the standardized audio data into an emotion enhancement module to generate acoustic features that match the target speech style through multiple rounds of feature fusion and autoregressive generation mechanism; The speech output unit is used to input the acoustic features into the speech synthesis module for decoding processing and output the predicted speech.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the voice cloning method based on emotion enhancement according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the voice cloning method based on emotion enhancement according to any one of claims 1 to 7 is implemented.

Images