A voice cloning method and device

Abstract

The invention relates to a voice cloning method, comprising: obtaining pronunciation data of each speaker that does not exceed a preset threshold value; each pronunciation data includes a speaker ID; extracting vector features of each pronunciation data; calculating all vector features corresponding to each speaker to obtain the speaker timbre features of the speaker; inputting the speaker timbre features of each speaker into a fixed part of a preset network, and the rest of the network initializes a preset reference model and updates the parameters of the reference model until a target reference model is obtained; obtaining the speaker timbre features of each current speaker; inputting the speaker timbre features of the current speaker into the network to synthesize speech with the timbre of the current speaker.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a voice cloning method and device. Background Art

[0002] In the prior art, in order to realize multi-person voice timbre cloning under low resources, the following methods are mainly used:

[0003] The first method is to train a model based on a benchmark model trained with large amounts of data and based on the voice data of a single person. Each speaker has his or her own set of models and all parameters.

[0004] The second method is to freeze some parameters of the network model based on the benchmark model trained with large amounts of data, so that each speaker has exclusive access to some parameters in the model and freeze parameter sharing.

[0005] The above two methods have the following problems respectively:

[0006] In the first method, each speaker has his own set of models and parameters. Although the similarity and naturalness of the cloned speech are good, the cost of the deployment stage increases sharply as the number of people increases. In the second method, for the case of frozen parameter training, although the deployment cost is reduced to a certain extent, the similarity and naturalness of the cloned speech are greatly reduced under low resources, and the timbre cloning fails.

[0007] Existing models find it difficult to extract the speaker's timbre characteristics when resources are low and the number of model parameters is limited, resulting in low similarity in the cloned speech. As the number of speakers increases, the storage and computing resources required in the deployment phase also increase dramatically. Summary of the invention

[0008] The purpose of the present invention is to provide a voice cloning method and device to solve the problems in the prior art.

[0009] In a first aspect, the present invention provides a voice cloning method, the method comprising:

[0010] Acquire pronunciation data of each speaker that does not exceed a preset threshold; each pronunciation data includes a speaker ID;

[0011] Extracting vector features of each pronunciation data;

[0012] Calculating all vector features corresponding to each speaker to obtain speaker timbre features of the speaker;

[0013] Inputting the timbre characteristics of each speaker into a fixed part of a preset network, and the rest of the network initializes a preset reference model and updates the parameters of the reference model until a target reference model is obtained;

[0014] Get the voice characteristics of the current speaker;

[0015] The timbre characteristics of the current speaker are input into the network to synthesize speech with the timbre of the current speaker.

[0016] In a possible implementation, the method further includes:

[0017] A benchmark model is trained based on a large-scale audio library; the large-scale audio library includes pronunciation data of a large number of speakers.

[0018] In a possible implementation, the preset threshold is 20.

[0019] In a possible implementation, calculating all vector features corresponding to each speaker to obtain the speaker timbre features of the speaker specifically includes:

[0020] The average of all vector features of each speaker is calculated to obtain the speaker timbre features corresponding to each speaker.

[0021] In a possible implementation manner, updating the parameters of the benchmark model until a target benchmark model is obtained specifically includes:

[0022] When the parameters are input into the benchmark model for training, until the benchmark model converges, a target benchmark model is obtained.

[0023] In one possible implementation,

[0024] The step of inputting the timbre characteristics of the current speaker into the network to synthesize the speech with the timbre of the current speaker specifically includes:

[0025] Determining the speaker timbre characteristics of the current speaker according to the current speaker ID;

[0026] According to the timbre characteristics of the current speaker, a speech with the timbre of the current speaker is synthesized.

[0027] In a second aspect, the present invention provides a voice cloning device, the device comprising:

[0028] An acquisition module, the acquisition module is used to acquire pronunciation data of each speaker that does not exceed a preset threshold; each pronunciation data includes a speaker ID;

[0029] An extraction module, the extraction module is used to extract vector features of each pronunciation data;

[0030] A calculation module, the calculation module is used to calculate all vector features corresponding to each speaker to obtain the speaker timbre features of the speaker;

[0031] A model training module, wherein the model training module is used to input the timbre characteristics of each speaker into a fixed part of a preset network, and the rest of the network is initialized by a preset reference model and updates the parameters of the reference model until a target reference model is obtained;

[0032] The acquisition module is also used to acquire the speaker's timbre characteristics of the current speaker;

[0033] A cloning module is used for inputting the timbre characteristics of a current speaker into a network to synthesize speech having the timbre of the current speaker.

[0034] In a third aspect, the present invention provides a chip system, including a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the voice cloning method described in any one of the first aspects is implemented.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to perform the voice cloning method described in any one of the first aspects.

[0036] In a fifth aspect, the present invention provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the voice cloning method as described in any one of the first aspects.

[0037] This application introduces the speaker's timbre features to conduct mixed training for multiple speakers. After the speaker's timbre features of multiple speakers are input into the benchmark model, a target benchmark model including multiple speakers is obtained. Subsequently, the current speaker's voice can be cloned based on the speaker's timbre features and the target benchmark model. This greatly reduces computing and storage resources and saves costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a voice cloning method according to the first embodiment of the present invention;

[0039] Figure 2 The second flowchart of the voice cloning method provided in the first embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of a voice cloning device provided in Embodiment 2 of the present invention;

[0041] Figure 4 A schematic diagram of the chip system structure provided in Embodiment 3 of the present invention;

[0042] Figure 5 A schematic diagram of a computer-readable storage medium provided in Embodiment 4 of the present invention;

[0043] Figure 6 A schematic diagram of a computer program product provided in accordance with Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0046] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0047] Embodiment 1

[0048] Figure 1 This is one of the flow charts of the voice cloning method provided in the first embodiment of the present invention. The method is applied in the scenario of multi-speaker cyclic training under low resources. Figure 1 As shown, the method comprises the following steps:

[0049] Step 110, obtaining pronunciation data of each speaker that does not exceed a preset threshold; each pronunciation data includes a speaker ID;

[0050] Specifically, the pronunciation data of each speaker is no more than 20, and then speaker_id is used to distinguish different speakers. For example, if the number of speakers is 8 and the pronunciation data of each speaker is 20, the total number of pronunciation data obtained is 8*20=160.

[0051] Step 120, extracting vector features of each pronunciation data;

[0052] Specifically, the vector feature is embedding. Embedding is to use a low-dimensional vector to represent an object, which can be a word, a product, a movie, etc.

[0053] Step 130, calculating all vector features corresponding to each speaker to obtain the speaker's timbre features;

[0054] Specifically, the average of all embeddings of each speaker is taken as the speaker timbre feature speaker_embedding of the speaker.

[0055] Step 140, inputting the timbre characteristics of each speaker into a fixed part of a preset network, and the rest of the network initializes a preset reference model and updates the parameters of the reference model until a target reference model is obtained;

[0056] Specifically, the network refers to a neural network, which is equivalent to an architecture. The benchmark model is a set of specific parameters for each node of the neural network, and the parameters will be updated during the training process. The multi-speaker speaker_embedding obtained in the previous step is connected to the encoder part of the network. This part of the parameters is fixed during the training process; the rest of the parameters of the Tacotron network are initialized with the benchmark model, and the learning rate and number of iterations are reduced to update the parameters of the model. The target benchmark model is mixed with multiple speaker information, and different speakers can be cloned. Subsequently, each time a different speaker's speaker_embedding is input, the speech of the corresponding speaker's timbre will be cloned.

[0057] Among them, in an optional implementation, such as Figure 2 As shown, before step 140, step 210 is also included:

[0058] Step 210, training a benchmark model based on a large-scale audio library; the large-scale audio library includes pronunciation data of a large number of speakers.

[0059] Specifically, a benchmark model is trained based on a large audio database containing multiple speakers. The Tacotron framework can be used. The framework is an encoder-decoder architecture. Speaker_embedding is introduced in the encoder part to represent the speaker's timbre characteristics, and a benchmark model for speech synthesis is obtained through training.

[0060] Among them, steps 110-140, or steps 110-140 combined with 210 are a training stage. The purpose of training in the training stage is to obtain a target benchmark model, which can be directly applied in the subsequent reasoning stage to obtain the speech of multiple speakers.

[0061] Step 150, obtaining the speaker's timbre characteristics of the current speaker;

[0062] Specifically, in the inference stage, for the scenario of multiple speakers, vector features of multiple speakers are obtained, and the vector features of each current speaker are averaged to obtain the speaker timbre features of the current speaker. The speaker timbre features have obvious inter-class separability. Separability means that the timbre of each speaker is very distinguishable. The reasons why this application achieves separability are: 1. Speaker vector features with good distinguishability are used; 2. The neural network fits the data during training, which will also help the separability between speakers.

[0063] Step 160, input the timbre characteristics of the current speaker into the network to synthesize speech with the timbre of the current speaker.

[0064] Specifically, during speech synthesis, if there are multiple speakers, you can arbitrarily select one from the current speakers as the current speaker, and input the current speaker's speaker_embedding into the network. Based on this, you can synthesize speech with high similarity and naturalness to the current speaker in the target benchmark model.

[0065] Among them, since the pronunciation data includes the speaker ID, it is only necessary to determine the speaker ID of the selected speaker, and then the speaker timbre characteristics of the selected speaker can be used as the input of the network based on the speaker ID, so that the voice of the current speaker can be directly cloned.

[0066] In order to take into account both the training speed and deployment cost of the target benchmark model, the number of speakers supported by the target benchmark model can be reasonably configured according to actual conditions to achieve a balance between resource loss and ease of use.

[0067] This application introduces the speaker's timbre features to perform mixed training on multiple speakers. After the speaker's timbre features of multiple speakers are input into the benchmark model, a target benchmark model including multiple speakers is obtained. Subsequently, the current speaker's voice can be cloned based on the speaker's timbre features and the target benchmark model. This greatly reduces computing and storage resources and saves costs.

[0068] Embodiment 2

[0069] Embodiment 2 of the present invention provides a voice cloning device, such as Figure 3As shown, the device includes: an acquisition module 310, an extraction module 320, a calculation module 330, a model training module 340 and a cloning module 350.

[0070] The acquisition module 310 is used to acquire the pronunciation data of each speaker that does not exceed a preset threshold; each pronunciation data includes a speaker ID;

[0071] The extraction module 320 is used to extract the vector features of each pronunciation data;

[0072] The calculation module 330 is used to calculate all the vector features corresponding to each speaker to obtain the speaker's timbre features;

[0073] The model training module 340 is used to input the timbre characteristics of each speaker into a fixed part of a preset network, and the rest of the network initializes a preset reference model and updates the parameters of the reference model until a target reference model is obtained;

[0074] The acquisition module 310 is also used to acquire the speaker's timbre characteristics of the current speaker;

[0075] The cloning module 350 is used to input the timbre characteristics of the current speaker into the network to synthesize speech with the timbre of the current speaker.

[0076] The model training module 340 is also used to train a benchmark model based on a large-scale audio library; the large-scale audio library includes pronunciation data of a large number of speakers.

[0077] Among them, the preset threshold is 20.

[0078] Furthermore, the calculation module 330 calculates all the vector features corresponding to each speaker to obtain the speaker's timbre features, which specifically includes: averaging all the vector features of each speaker to obtain the speaker's timbre features corresponding to each speaker.

[0079] Furthermore, the model training module 340 updates the parameters of the benchmark model until the target benchmark model is obtained, which specifically includes: when the parameters are input into the benchmark model for training, until the benchmark model converges, the target benchmark model is obtained.

[0080] Further, the speaker timbre characteristics of the selected speaker include the selected speaker ID; the cloning module 350 inputs the speaker timbre characteristics of the current speaker into the network, and synthesizes the speech with the timbre of the current speaker specifically includes: determining the speaker timbre characteristics of the current speaker according to the current speaker ID; synthesizing the speech with the timbre of the current speaker according to the timbre characteristics of the current speaker. .

[0081] The device provided in the second embodiment of the present invention can execute the method steps in the above-mentioned method embodiment 1, and its implementation principle and technical effects are similar, which will not be repeated here.

[0082] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the determination module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0083] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0084] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0085] Embodiment 3

[0086] Embodiment 3 of the present invention provides a chip system, such as Figure 4 As shown, a processor is included, and the processor is coupled to a memory, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, any one of the voice cloning methods provided in the first embodiment is implemented.

[0087] Embodiment 4

[0088] Embodiment 4 of the present invention provides a computer-readable storage medium, such as Figure 5 As shown, it includes a program or instruction, and when the program or instruction is run on a computer, any one of the voice cloning methods provided in the first embodiment is implemented.

[0089] Embodiment 5

[0090] Embodiment 5 provides a computer program product including instructions, such as Figure 6 As shown, when the computer program product runs on a computer, the computer is enabled to execute any one of the voice cloning methods provided in the first embodiment.

[0091] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0092] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0093] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A voice cloning method, characterized in that: The method comprises: Acquire pronunciation data of each speaker that does not exceed a preset threshold; each pronunciation data includes a speaker ID; Extracting vector features of each pronunciation data; Calculating all vector features corresponding to each speaker to obtain speaker timbre features of the speaker; Inputting the timbre characteristics of each speaker into a fixed part of a preset neural network, and the rest of the neural network initializes a preset reference model and updates the parameters of the preset reference model until a target reference model is obtained; Get the voice characteristics of the current speaker; Inputting the timbre characteristics of the current speaker into the target reference model to synthesize speech with the timbre of the current speaker; The calculation of all vector features corresponding to each speaker to obtain the speaker timbre features of the speaker specifically includes: The average of all vector features of each speaker is calculated to obtain the speaker timbre features corresponding to each speaker; The step of inputting the timbre characteristics of the current speaker into the target reference model to synthesize the speech with the timbre of the current speaker specifically includes: Determining the speaker timbre characteristics of the current speaker according to the current speaker ID; According to the timbre characteristics of the current speaker, a speech with the timbre of the current speaker is synthesized.

2. The method according to claim 1, characterized in that: The method also includes: A benchmark model is trained based on a large-scale audio library; the large-scale audio library includes pronunciation data of a large number of speakers.

3. The method according to claim 2, characterized in that The preset threshold is 20.

4. The method according to claim 1, characterized in that The updating of the parameters of the preset benchmark model until the target benchmark model is obtained specifically includes: When the speaker's timbre features are input into the preset benchmark model for training until the benchmark model converges, a target benchmark model is obtained.

5. A voice cloning device, characterized in that: The device comprises: An acquisition module, the acquisition module is used to acquire pronunciation data of each speaker that does not exceed a preset threshold; each pronunciation data includes a speaker ID; An extraction module, the extraction module is used to extract vector features of each pronunciation data; A calculation module, the calculation module is used to calculate all vector features corresponding to each speaker to obtain the speaker timbre features of the speaker; A model training module, wherein the model training module is used to input the timbre characteristics of each speaker into a fixed part of a preset neural network, and the rest of the neural network is initialized by a preset reference model and updates the parameters of the preset reference model until a target reference model is obtained; The acquisition module is also used to acquire the speaker's timbre characteristics of the current speaker; A cloning module, the cloning module is used to input the timbre characteristics of the speaker of the current speaker into the target reference model to synthesize speech with the timbre of the current speaker; The calculation module calculates all the vector features corresponding to each speaker to obtain the speaker timbre features of the speaker, specifically including: averaging all the vector features of each speaker to obtain the speaker timbre features corresponding to each speaker; The cloning module inputs the timbre characteristics of the current speaker into the target reference model, and synthesizes the speech with the timbre of the current speaker, specifically including: Determining the speaker timbre characteristics of the current speaker according to the current speaker ID; According to the timbre characteristics of the current speaker, a speech with the timbre of the current speaker is synthesized.

6. A chip system, characterized in that: The invention comprises a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the voice cloning method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to execute the voice cloning method according to any one of claims 1 to 4.

8. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the voice cloning method according to any one of claims 1 to 4.

Images