Rhythm control method in voice timbre conversion, electronic device and storage medium

Abstract

The application discloses a rhythm control method in voice timbre conversion, comprising the following steps: obtaining a source voice semantic vector and a source voice mel spectrum of source voice; obtaining a source voice rhythm vector of the source voice after perturbing the source voice; obtaining a reference voice semantic vector, a reference voice rhythm vector and a reference voice mel spectrum of reference voice; masking part of the mel spectrum; supplementing the source voice semantic vector and the source voice rhythm vector into the output voice mel spectrum, obtaining timbre information through the mel spectrum of the reference voice, outputting the output voice mel spectrum, and completing timbre conversion. The application uses the training mode of masking and reconstruction, and can avoid the demand for paired data and clean data. Meanwhile, the application can realize timbre conversion in a simple way.

Description

Technical Field

[0001] This invention relates primarily to the field of timbre conversion, and more particularly to a rhythm control method, electronic device, and storage medium for speech timbre conversion. Background Technology

[0002] For zero-sample speech-to-speech tasks, a single sentence from the target speaker's voice is typically used as a reference to generate speech based on that person's vocal timbre. However, many speech-to-speech tasks only focus on whether the timbre conversion is good enough, without considering whether the converted speech retains the original rhythm. Maintaining the original rhythm is crucial in many scenarios. For example, in online live streaming or video scenarios, we need to ensure that the converted speech is synchronized with the speaker's mouth and body movements—that is, to maintain the original speaker's rhythm. Summary of the Invention

[0003] Given that existing technologies do not address whether the converted speech retains its original rhythm, the technical problems to be solved by this invention include:

[0004] How to design a rhythm control method in speech timbre conversion to maintain the original speaker's rhythm.

[0005] To achieve the above objectives, the present invention provides a rhythm control method in speech timbre conversion, comprising: obtaining the source speech semantic vector and the source speech Mel spectrum;

[0006] After perturbating the source speech, the source speech rhythm vector is obtained;

[0007] Obtain the reference speech semantic vector, reference speech rhythm vector, and reference speech Mel spectrum;

[0008] Mel spectrum that masks part of the speech;

[0009] The Mel spectrum of the output speech is constructed by supplementing the semantic vector and rhythm vector of the source speech into the Mel spectrum of the output speech, and the timbre information is obtained by using the Mel spectrum of the reference speech.

[0010] Output the speech spectrum to complete the timbre conversion.

[0011] Furthermore, the acquisition of the source speech semantic vector specifically involves:

[0012] The source speech is input into the HuBERT or Wav2Vec model. The HuBERT or Wav2Vec model uses the K-means algorithm to discretize the information in the source speech and decouple the content information in the speech. The Embedding Table structure is used to transform the content information into a continuous source speech semantic vector.

[0013] Furthermore, the perturbation of the source speech specifically involves:

[0014] Use the sox tool to raise or lower the pitch of the source speech.

[0015] Furthermore, the acquisition of the rhythm vector of the perturbed source speech specifically involves:

[0016] The source speech rhythm vector is extracted from the source speech using the Emotion2Vec model.

[0017] Furthermore, the acquisition of the source speech Mel spectrum specifically involves:

[0018] Using the vocos tool, the source speech is input to obtain the source speech Mel spectrum.

[0019] Furthermore, the acquisition of the source speech Mel spectrum of the reference speech specifically involves:

[0020] Using the vocos tool, the reference speech is input to obtain the reference speech Mel spectrum.

[0021] Furthermore, the masking of the source speech Mel spectrum specifically involves randomly masking a portion of the continuous source speech Mel spectrum, such that the remaining source speech Mel spectrum is between 2 and 3 seconds.

[0022] Furthermore, the masked portion of the source speech Mel spectrum also includes replacing the masked portion of the source speech Mel spectrum with random noise.

[0023] An electronic device includes a memory, a processor, and a program stored in the memory, the program being configured to be executed by the processor, wherein the processor, when executing the program, implements the rhythm control method in speech timbre conversion as described above.

[0024] A storage medium storing a computer program that, when executed by a processor, implements the rhythm control method in speech timbre conversion as described above.

[0025] Compared with existing technical solutions, the technical advantages of the present invention are as follows:

[0026] This invention uses an emotion recognition-related pre-trained model, Emotion2Vec, to extract speech rhythm information. Simultaneously, the pitch of the speech is perturbed before being input into the pre-trained model to prevent the leakage of timbre-related information.

[0027] This invention uses a masking and reconstruction method for training, which avoids the need for paired and clean data. It also allows for very simple timbre conversion.

[0028] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0029] Figure 1 This is a system structure diagram of the present invention;

[0030] Figure 2 This is a training diagram of the present invention. Detailed Implementation

[0031] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0032] A rhythm control method in speech timbre conversion, such as Figure 1 and Figure 2 As shown, this includes obtaining the source speech semantic vector and the source speech Mel spectrum;

[0033] After perturbating the source speech, the source speech rhythm vector is obtained;

[0034] Obtain the reference speech semantic vector, reference speech rhythm vector, and reference speech Mel spectrum;

[0035] Mel spectrum that masks part of the speech;

[0036] The Mel spectrum of the output speech is constructed by supplementing the semantic vector and rhythm vector of the source speech into the Mel spectrum of the output speech, and the timbre information is obtained by using the Mel spectrum of the reference speech.

[0037] Output the speech spectrum to complete the timbre conversion.

[0038] The specific steps for obtaining the source speech semantic vector are as follows:

[0039] The source speech is input into the HuBERT or Wav2Vec model. The HuBERT or Wav2Vec model uses the K-means algorithm to discretize the information in the source speech and decouple the content information in the speech. The Embedding Table structure is used to transform the content information into a continuous source speech semantic vector.

[0040] The perturbation of the source speech specifically involves:

[0041] Use the sox tool to raise or lower the pitch of the source speech.

[0042] The specific steps for obtaining the rhythm vector of the perturbated source speech are as follows:

[0043] The source speech rhythm vector is extracted from the source speech using the Emotion2Vec model.

[0044] The specific steps for obtaining the Mel spectrum of the source speech are as follows:

[0045] Using the vocos tool, the source speech is input to obtain the source speech Mel spectrum.

[0046] The specific steps for obtaining the Mel spectrum of the source speech of the reference speech are as follows:

[0047] Using the vocos tool, the reference speech is input to obtain the reference speech Mel spectrum.

[0048] The masking of the source speech Mel spectrum specifically involves randomly masking a portion of the continuous source speech Mel spectrum, such that the remaining source speech Mel spectrum is between 2 and 3 seconds.

[0049] The process of masking the source speech Mel spectrum also includes replacing the masked portion of the source speech Mel spectrum with random noise.

[0050] like Figure 1 As shown, we use two pre-trained models to decouple information. HuBERT / Wav2Vec decouples the content information in speech through K-means discretization. Then, we use an embedding table to transform the K-means discretized content information into a continuous semantic vector. For decoupling speech rhythm information, we use the Emotion2Vec model. This is an emotion recognition model. We believe that human emotions are mostly reflected through the rhythm of a speaker, so the features extracted from the Emotion2Vec model will contain speech rhythm information. In addition, we perform pitch perturbation on the speech before sending it to Emotion2Vec. The purpose of this is to change the timbre of the input speech to prevent the extracted rhythm vector (ProsodyEmbedding) from containing timbre information.

[0051] like Figure 2As shown, during training, our system receives three inputs: a semantic embedding containing the speech content, a prosody embedding containing the prosodic variation, and a mel spectrogram extracted from the current speech, containing all the speech information. Additionally, we mask out parts of the mel spectrogram. We hope our generative model can reconstruct these masked portions. For reconstructing the masked mel spectrogram, content information can be obtained from the current position's semantic embedding, rhythm information from the current position's prosody embedding, and timbre information, since the entire sentence belongs to the same person, can be obtained from the speech context.

[0052] So how do we use our trained model to perform timbre conversion, such as... Figure 1 As shown, we extract three inputs: reference speech, source speech, and mel spectrum. Additionally, we directly mask the mel spectrum corresponding to the source speech. Thus, when reconstructing the source speech, the timbre-related information must be obtained from the mel spectrum of the reference speech, thereby achieving timbre conversion.

[0053] The table below shows the timbre conversion results of our system (ICL-VC) and other systems. In the method described above, we propose using Prosody Emb / Emotion Emb to extract rhythmic information. Besides this method, we can also simply extract pitch and energy information to reflect the rhythmic information of the input. However, we found that the speaker similarity (SECS) obtained after timbre conversion based on pitch and energy information is lower than that obtained using our proposed Emotion Emb method. Furthermore, the system based on our Emotion Emb method generates a higher naturalness MOS. Simultaneously, our system also performs better than other speech conversion systems (YourTTS, RefXVC).

[0054]

[0055] Table 1: Speech Conversion Results

[0056] in:

[0057] SECS: Speaker embedding cosine similarity, represents the similarity between the converted timbre of the system-generated speech and the timbre of the reference speech. The higher the similarity, the better.

[0058] CER: Character Error Rate, which represents the error rate in speech recognition of the converted voice. The lower the error rate, the higher the intelligibility of the speech.

[0059] Pitch Corr: pitch correlation, which represents the similarity between the pitch changes of the converted speech and the original speech. The higher this index, the better.

[0060] Energy Corr: energy correlation, which represents the similarity between the energy changes of the converted speech and the original speech. The higher this index, the better.

[0061] Prosody MOS: This is a score given by the supervisor regarding the similarity in rhythm between the converted voice and the original voice. The higher the score, the more similar the voice.

[0062] Naturalness MOS: This is a subjective score that assesses whether the voice sounds natural after timbre conversion; the higher the score, the better.

[0063] As we can see from our system, ICL-VC+Emotion Emb is superior to ICL-VC+.

[0064] Pitch & Energy performed better on the SECS metric, and also had higher Prosody MOS and Naturalness MOS scores.

[0065] An electronic device includes a memory, a processor, and a program stored in the memory, the program being configured to be executed by the processor, wherein the processor, when executing the program, implements the rhythm control method in speech timbre conversion as described above.

[0066] A storage medium storing a computer program that, when executed by a processor, implements the rhythm control method in speech timbre conversion as described above.

[0067] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for controlling tempo in voice timbre conversion, characterized by: The method comprises the following steps: obtaining a source speech semantic vector and a source speech mel spectrum of a source speech; obtaining a source speech rhythm vector of the source speech after perturbing the source speech; obtaining a reference speech semantic vector, a reference speech rhythm vector and a reference speech mel spectrum of a reference speech; masking part of the mel spectrum; establishing an output speech mel spectrum, supplementing the source speech semantic vector and the source speech rhythm vector into the output speech mel spectrum, and obtaining timbre information through the mel spectrum of the reference speech; outputting the output speech mel spectrum to complete the timbre conversion; The method comprises the following steps: extracting a source speech rhythm vector in the source speech through an Emotion2Vec model; The method comprises the following steps: randomly masking part of the continuous source speech mel spectrum, and making the remaining source speech mel spectrum between 2-3s.

2. The method of tempo control in voice timbre conversion according to claim 1, characterized in that: The method comprises the following steps: The method comprises the following steps:

3. The method of tempo control in voice timbre conversion according to claim 1, wherein, inputting the source speech into a HuBERT model or a Wav2Vec model, using a K-means algorithm to discretize the information in the source speech, and decoupling the content information in the speech; using an Embedding Table structure to convert the content information into a continuous source speech semantic vector. The method comprises the following steps:

4. The method of tempo control in voice timbre conversion according to claim 1, wherein, using a sox tool to raise or lower the pitch of the source speech. The method comprises the following steps:

5. The method of tempo control in voice timbre conversion according to claim 1, wherein, using a vocos tool to input the source speech to obtain a source speech mel spectrum. The method comprises the following steps:

6. An electronic device, comprising: using a vocos tool to input the reference speech to obtain a reference speech mel spectrum.

7. A storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the rhythm control method in the voice timbre conversion according to any one of claims 1-5. The computer program is executed by the processor to implement the rhythm control method in the voice timbre conversion according to any one of claims 1-5.

Images