Speech conversion model generation method and device

Through the speech conversion model based on neural network, the loss is calculated and the model parameters are updated using the feature embedding vectors of the speech of the original speaker and the target speaker, the problem of insufficient naturalness and synchronization of real-time speech conversion in the existing technology is solved, and the reality is realized speech conversion and retention of rhythm and emotion.

CN114141269BActive Publication Date: 2024-11-22CHINA TELECOM CORP LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202010808769.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-12
Publication Date
2024-11-22
Estimated Expiration
2040-08-12

AI Technical Summary

Technical Problem

Existing voice conversion technologies cannot achieve real-time, natural and realistic voice conversion, especially when synchronous synthesis is required, and users need to have an understanding of multiple parameters.

Method used

A speech conversion model based on neural network is adopted, and the final speech conversion model is generated by feature extraction and embedding vector calculations for the original speaker speech and the target speaker speech, and the model parameters are backpropagated until the loss is less than the threshold.

Benefits of technology

It realizes that without parameter settings, the original speaker's tone is naturally and realistically converted to the target speaker's tone, and retains rhythm, emotion and speech speed, realizing basic real-time voice conversion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114141269B_ABST
    Figure CN114141269B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and device for generating a speech conversion model. A speech conversion model generation method is provided, which is used to train a speech conversion model based on a neural network, and comprises the following steps: a. using the speech conversion model to convert the original speaker's speech; b. extracting the audio features of the converted speech and the target speaker's speech respectively; c. inputting the audio features of the converted speech and the target speaker's speech into a pre-trained speaker feature encoder respectively, and obtaining the original speaker feature embedding vector and the target speaker feature embedding vector respectively; d. calculating the loss between the original speaker feature embedding vector and the target speaker feature embedding vector; e. back-propagating the loss to the speech conversion model, thereby updating the parameters of the speech conversion model; f. using the updated speech conversion model, repeating the above steps a to e until the loss is less than a predetermined threshold or the number of iterations is greater than a predetermined number.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present technology relates to the technical field of speech conversion model generation method and device. Background Art

[0002] Real-time voice conversion technology has a wide range of application scenarios. It means retaining the original speaker's rhythm, emotion, speaking speed and other characteristics, and converting them into the voice of the designated speaker in real time while keeping the content of the speech consistent. For example, players can "change their voices" while playing games to make voice interactions in games more entertaining. In telephone customer service scenarios, the customer service voice can be made softer to enhance the user experience. In the live broadcast field, the anchor uses voice conversion software to transform into the voice of a "star" to attract users to watch.

[0003] Currently, the common voice changing algorithms are: 1) based on data signal processing to adjust the pitch and timbre. The adjusted sound has limitations and is not realistic and natural enough. In addition, the user is required to have a certain understanding of various parameters; 2) based on speech synthesis methods, the original audio is converted into audio features and then synthesized into the target sound through speech synthesis methods. The sound synthesized by this method is more natural, but this method essentially requires receiving a complete sentence audio before conversion, and cannot achieve basic real-time synchronous synthesis.

[0004] Therefore, a new technology needs to be proposed to solve one or more problems in the above-mentioned prior art. Summary of the invention

[0005] According to one aspect of the present disclosure, a method for generating a speech conversion model is provided, which is used to train a speech conversion model based on a neural network, and includes the following steps: a. using the speech conversion model to convert the original speaker's speech; b. extracting the audio features of the converted speech and the target speaker's speech respectively; c. inputting the audio features of the converted speech and the target speaker's speech into a pre-trained speaker feature encoder respectively, and obtaining an original speaker feature embedding vector and a target speaker feature embedding vector respectively; d. calculating the loss between the original speaker feature embedding vector and the target speaker feature embedding vector; e. back-propagating the loss to the speech conversion model, thereby updating the parameters of the speech conversion model; f. using the updated speech conversion model, repeating the above steps a to e until the loss is less than a predetermined threshold or the number of iterations is greater than a predetermined number.

[0006] According to another aspect of the present disclosure, a real-time speech conversion method is provided, comprising: receiving speech of an original speaker; and converting the speech of the original speaker using a speech conversion model generated according to the above method.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, comprising executable instructions, and when the executable instructions are executed by an information processing device, the information processing device is caused to execute the above methods.

[0008] According to another aspect of the present disclosure, a speech conversion model generation device is provided, which is used to train a speech conversion model based on a neural network, comprising: one or more processors; and a memory on which executable instructions are stored, and when the executable instructions are executed by the one or more processors, the one or more processors are caused to: a. use the speech conversion model to convert the original speaker's speech; b. extract the audio features of the converted speech and the target speaker's speech respectively; c. input the audio features of the converted speech and the target speaker's speech into a pre-trained speaker feature encoder respectively, and obtain the original speaker feature embedding vector and the target speaker feature embedding vector respectively; d. calculate the loss between the original speaker feature embedding vector and the target speaker feature embedding vector; e. back-propagate the loss to the speech conversion model, thereby updating the parameters of the speech conversion model; f. use the updated speech conversion model to repeat the above processes a to e until the loss is less than a predetermined threshold or the number of iterations is greater than a predetermined number.

[0009] According to another aspect of the present disclosure, a real-time speech conversion device is provided, comprising: one or more processors; and a memory having executable instructions stored thereon, wherein when the executable instructions are executed by the one or more processors, the one or more processors are enabled to: receive the speech of an original speaker; and convert the speech of the original speaker using a speech conversion model generated by the above-mentioned device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flowchart illustrating a method for generating a speech conversion model according to an embodiment of the present disclosure.

[0011] Figure 2 is an example of a speech conversion model according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0013] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0014] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0015] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0016] As mentioned above, the two main existing voice changing algorithms have their own limitations. In order to solve one or more of the above technical problems existing in the prior art, the inventor of the present application proposes a new technical concept: using a speech conversion model based on a neural network to convert speech, and using the original speaker's speech and the target speaker's speech to train the speech conversion model.

[0017] Figure 1 1 is a flowchart illustrating a method 100 for generating a speech conversion model according to an embodiment of the present disclosure, wherein the method is used to train an untrained neural network-based speech conversion model to generate a final speech conversion model.

[0018] In method 100, first, the original speaker's speech is converted using a speech conversion model (step 101). In some embodiments, the original speaker's speech can be received and converted. In some embodiments, the original speaker's speech stored on a server or cloud can be converted.

[0019] In some embodiments, during the conversion, the original speaker's speech needs to be divided into a plurality of frames having a first time length (e.g., 30 ms (milliseconds)), and one or more frames are input into the speech conversion model each time. In some embodiments, when the speech is divided into frames, a sliding window having a second time length (e.g., 10 ms) can be used so that there is an overlapping time length between adjacent frames.

[0020] In some embodiments, one or more frames of the original speaker's speech may be input into the speech conversion model each time, and the speech conversion model may be used to convert the one or more frames of the original speaker's speech into speech data of one frame. By repeating this process, the original speaker's speech with a predetermined time length (e.g., about half an hour) may be converted into speech data of multiple frames that are continuous in time. It should be noted that the above-mentioned speech conversion may be performed substantially in real time.

[0021] As an example, when the first time length is 30ms and the second time length is 10ms, the original speaker's speech can be divided into the following frames: frame 1 (time interval 0-30ms), frame 2 (time interval 10-40ms), frame 3 (time interval 20-50ms), frame 4 (time interval 30-60ms), frame 5 (time interval 40-70ms), frame 6 (time interval 50-80ms), frame 7 (time interval 60-90ms), frame 8 (time interval 70-100ms)...

[0022] As an example, 5 frames of the original speaker's speech can be input into the speech conversion model each time, and the next group of 5 frames can be input three frames at a time. For example, the first time, the 1st to 5th frames are input, the second time, the 4th to 8th frames are input... In some embodiments, the speech can be output every three frames. Thus, the time interval of the first frame of the converted speech is 20 to 50ms, and the time interval of the second frame is 50 to 80ms, so that the frames of the converted speech can be ensured to be continuous in time. In addition, when multiple frames of the original speaker's speech are input each time, the input speech can have sufficient context information, thereby improving the accuracy of speech conversion.

[0023] It should be noted that all the above numerical values ​​are exemplary, and the technology disclosed in the present invention is not limited to these specific numerical values.

[0024] After step 101, audio features of the converted speech and the target speaker's speech may be extracted respectively (step 102). Similar to step 101, the target speaker's speech may be received and feature extraction may be performed on it. In some embodiments, feature extraction may be performed on the target speaker's speech stored on a server or cloud. In some embodiments, feature extraction may be performed on the speech using a time window of a first time length (e.g., 30 ms (milliseconds)) and a sliding window of a second time length (e.g., 10 ms). In some embodiments, the target speaker's speech may have a predetermined time length (e.g., half an hour).

[0025] An audio feature refers to any feature that can characterize the characteristics of a speaker's speech. In some embodiments, the audio feature may include, for example, short-time energy, zero-crossing rate, and Mel Frequency Cepstral Coefficents (MFCC).

[0026] In some embodiments, the speech content of the original speaker's speech and the target speaker's speech may be the same, and may have the same predetermined time length (eg, half an hour).

[0027] After step 102, the audio features of the converted speech and the target speaker speech extracted in step 102 may be respectively input into a pre-trained speaker feature encoder to obtain an original speaker feature embedding vector and a target speaker feature embedding vector (step 103).

[0028] In some embodiments, the speaker feature encoder may be a speech feature encoder based on a neural network. In the present disclosure, the speech feature encoder is used to convert the audio features of the speaker into a speaker feature embedding vector, in which one or more speaker speech features are embedded. Using the converted speaker feature embedding vector, the difference in speech features of different speakers can be more easily quantified.

[0029] In some embodiments, before using the speaker feature encoder, the speaker feature encoder can be pre-trained using a large amount of human speech.

[0030] In some embodiments, a neural network-based autoencoder including an encoder and a decoder can be pre-trained using a large amount of any person's speech, and the encoder part of the pre-trained autoencoder can be used as a speaker feature encoder.

[0031] After step 103, the loss between the original speaker feature embedding vector and the target speaker feature embedding vector can be calculated (step 104). In some embodiments, the loss can be any function that can be used to evaluate the difference between two vectors. In some embodiments, the loss can be, for example, a square error loss, an absolute error loss, a Huber loss, or a cross entropy loss. Those skilled in the art can select a suitable loss according to the specific circumstances when implementing the present technology. For example, for certain structures or types of neural networks, a cross entropy loss can be selected to speed up the learning process of the neural network. In some embodiments, the loss can be calculated for the speaker feature embedding vectors and the target speaker feature embedding vectors of all frames over the predetermined time length.

[0032] After step 104, the loss may be back-propagated into the speech conversion model, thereby updating the parameters of the speech conversion model (step 105).

[0033] Using the updated speech conversion model, the above steps 101 to 105 may be repeated until the loss is less than a predetermined threshold or the number of iterations is greater than a predetermined number, thereby generating a final speech conversion model.

[0034] The final voice conversion model can directly convert the original speaker's timbre into the target speaker's timbre naturally and realistically, and can preserve the original speaker's rhythm, emotion, and speaking speed. In addition, when the user uses the voice conversion model, no parameter settings are required.

[0035] In the present disclosure, the speech conversion model can be a model based on any type of neural network (e.g., a deep neural network, a time-delay neural network, a convolutional neural network, a recurrent neural network, and a combination of multiple types of neural networks). In some embodiments, the present disclosure can use a deep neural network based on a time-delay neural network (TDNN) structure.

[0036] Figure 2 is a diagram showing the neural network structure of a speech conversion model according to an embodiment of the present disclosure, which is based on a 3-layer TDNN structure. Obviously, the depth of the neural network in the present disclosure is not subject to any restrictions. For this neural network, for example, the first time length (frame length) is 30ms, and the second time length (sliding window size) is 10ms. In the example structure of the neural network, the converted speech signal is output at t, t+3 (representing t+30ms, similarly below), t+6... respectively, and the conversion signal at time t needs to be delayed to t+8 for output, that is, the system delay is 80ms.

[0037] In the present disclosure, by designing the structure of the neural network, the time delay of the converted speech of the original speaker's speech does not exceed 200ms. Therefore, the user will not feel obvious delay, so that substantially real-time speech conversion can be achieved.

[0038] In some embodiments, a real-time speech conversion method may be provided, comprising: receiving the speech of an original speaker; and converting the speech of the original speaker using the speech conversion model generated according to the above method. As described above, by performing the conversion, the speech of the original speaker can be converted into the speech of the timbre of the target speaker.

[0039] In some embodiments, the speech conversion model generation process of the present disclosure may be implemented in a speech conversion model generation device. The speech conversion model generation device may include a processing circuit, which may be configured to perform the speech conversion model generation process.

[0040] In some embodiments, the speech conversion model generation device may include one or more processors and a memory, on which executable instructions may be stored, which, when executed by the one or more processors, cause the one or more processors to perform the above-mentioned speech conversion model generation process.

[0041] In some embodiments, a real-time speech conversion device may be provided, comprising: one or more processors; and a memory having executable instructions stored thereon, wherein when the one or more processors are executed, the one or more processors are enabled to: receive the speech of an original speaker; and convert the speech of the original speaker using a speech conversion model generated by the speech conversion model generation device.

[0042] In one embodiment, the present disclosure may provide a computer-readable storage medium comprising executable instructions, which, when executed by an information processing device (e.g., a device such as a computer or a smart phone), causes the information processing device to perform the above-mentioned speech conversion model generation processing or real-time speech conversion processing.

Claims

1. A method for generating a timbre conversion model, the method being used to train a timbre conversion model based on a neural network, comprising the following steps: a. converting the original speaker's voice using the timbre conversion model to output continuous speech frames, wherein the original speaker's voice and the target speaker's voice have the same speech content and have a predetermined time length; b. extracting audio features of the converted speech and the target speaker's speech respectively; c. Input the converted speech and the audio features of the target speaker's speech into the pre-trained speaker feature encoder to obtain the original speaker feature embedding vector and the target speaker feature embedding vector respectively; d. Calculate the loss used to evaluate the difference between the original speaker feature embedding vector and the target speaker feature embedding vector; e. back-propagating the loss to the timbre conversion model, thereby updating the parameters of the timbre conversion model; f. Using the updated timbre conversion model, repeat the above steps a to e until the loss is less than a predetermined threshold or the number of iterations is greater than a predetermined number.

2. The method according to claim 1, wherein step a comprises: The timbre conversion model is used to convert the original speaker's voice into speech data of a plurality of frames that are continuous in time.

3. The method according to claim 1, wherein: The audio features include short-time energy, zero-crossing rate and Mel-frequency cepstral coefficients.

4. The method according to claim 1, wherein: The timbre conversion model is a deep neural network based on a time-delay neural network structure.

5. The method according to claim 1, wherein: The time delay of the converted speech from the original speaker's speech does not exceed 200 ms.

6. A real-time voice conversion method, comprising: Receive the original speaker's voice; and The original speaker's speech is converted using the timbre conversion model generated by the method described in any one of claims 1-5.

7. A device for generating a timbre conversion model, the device being used for training a timbre conversion model based on a neural network, comprising: one or more processors; and a memory having executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to: a. converting the original speaker's voice using the timbre conversion model to output continuous speech frames, wherein the original speaker's voice and the target speaker's voice have the same speech content and have a predetermined time length; b. extracting audio features of the converted speech and the target speaker's speech respectively; c. Input the converted speech and the audio features of the target speaker's speech into the pre-trained speaker feature encoder to obtain the original speaker feature embedding vector and the target speaker feature embedding vector respectively; d. Calculate the loss used to evaluate the difference between the original speaker feature embedding vector and the target speaker feature embedding vector; e. back-propagating the loss to the timbre conversion model, thereby updating the parameters of the timbre conversion model; f. Using the updated timbre conversion model, repeat the above processes a to e until the loss is less than a predetermined threshold or the number of iterations is greater than a predetermined number.

8. The apparatus according to claim 7, wherein the process a comprises: The timbre conversion model is used to convert the original speaker's voice into temporally continuous voice data.

9. The device according to claim 7, wherein: The audio features include short-time energy, zero-crossing rate and Mel-frequency cepstral coefficients.

10. The device according to claim 7, wherein: The timbre conversion model is a deep neural network based on a time-delay neural network structure.

11. The device according to claim 7, wherein: The time delay of the converted speech from the original speaker's speech does not exceed 200 ms.

12. A real-time voice conversion device, comprising: one or more processors; and a memory having executable instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to: receiving the original speaker's speech; and The original speaker's speech is converted using the timbre conversion model generated by the apparatus according to any one of claims 7 to 11.

13. A computer-readable storage medium comprising executable instructions, which, when executed by an information processing device, cause the information processing device to execute the method according to any one of claims 1 to 5 or 6.

Citation Information

Patent Citations

  • Voice conversion method and device, equipment and storage medium

    CN109637551A

  • Speaker segmentation labeling method and device based on long-short-term memory neural network

    CN110910891A

  • Training method and device of neural network for extracting speaker embedding features

    CN111191787A

  • Voice conversion method, system and device and storage medium

    CN111247584A

  • Voice conversion method, device and equipment and storage medium

    CN111247585A