Speech cloning method and device based on autoregression model, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of financial science and technology, and discloses a voice cloning method, device and equipment based on an autoregression model, and a storage medium, and the method comprises the steps: carrying out the self-supervision feature extraction of a preset reference voice, obtaining a first feature vector, carrying out the style feature extraction of the preset reference voice, and obtaining a second feature vector; performing text feature extraction on the original text to obtain a third feature vector; based on the first feature vector, the second feature vector and the third feature vector, determining simulation voice corresponding to the original text; obtaining a loss value between the simulated voice and a preset reference voice, and training an autoregression model based on the loss value; and through the trained autoregression model, carrying out self-supervised feature extraction on the current reference voice to obtain a fourth feature vector, and processing the fourth feature vector to generate a cloned voice corresponding to the current text. According to the invention, personalized clone voice can be generated, and application requirements can be met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of financial technology. In particular, the present invention relates to a voice cloning method, device, equipment and storage medium based on an autoregressive model. Background Art

[0002] With the continuous advancement of artificial intelligence technology, voice cloning technology has ushered in unprecedented development opportunities and has shown broad application potential in multiple fields.

[0003] Due to the limitation of learning mechanism, voice cloning technology is difficult to capture self-supervision features, which are useful information automatically extracted from a large amount of unlabeled data. Since it is impossible to learn self-supervision features, the cloned voices generated by voice cloning technology lack personalization, which limits the application scenarios of voice cloning technology. Therefore, the cloned voices generated by existing voice cloning methods lack personalization and are not conducive to meeting application requirements. Summary of the invention

[0004] The present invention provides a voice cloning method, device, computer equipment and storage medium based on an autoregressive model, so as to solve the technical problem that the cloned voice generated by the existing voice cloning method lacks personalization.

[0005] In a first aspect, a voice cloning method based on an autoregressive model is provided, comprising:

[0006] Acquire an original text and a preset reference voice corresponding to the original text;

[0007] By using an autoregressive model, in a self-supervised learning manner, self-supervised feature extraction is performed on the preset reference speech to obtain a first feature vector, style feature extraction is performed on the preset reference speech to obtain a second feature vector, and text feature extraction is performed on the original text to obtain a third feature vector;

[0008] Determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector;

[0009] Obtaining a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value;

[0010] When the loss value meets the preset condition, obtaining the trained autoregressive model;

[0011] The current text and the current reference speech are obtained, and self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text.

[0012] Furthermore, the method of performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through the autoregressive model to obtain a first feature vector, performing style feature extraction on the preset reference speech to obtain a second feature vector, and performing text feature extraction on the original text to obtain a third feature vector includes:

[0013] Performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through a residual encoder in an autoregressive model to obtain a first feature vector;

[0014] The style feature of the preset reference speech is extracted by the style encoder in the autoregressive model to obtain a second feature vector; and the text feature of the original text is extracted by the priori encoder in the autoregressive model to obtain a third feature vector.

[0015] Further, the determining the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector includes:

[0016] Input the first feature vector, the second feature vector, and the third feature vector into a residual decoder in an autoregressive model;

[0017] The first feature vector, the second feature vector, and the third feature vector are processed by a residual decoder in an autoregressive model to generate simulated speech corresponding to the original text.

[0018] Furthermore, the acquiring a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value, includes:

[0019] Processing the simulated speech and the preset reference speech through a posterior encoder in an autoregressive model to obtain audio features of the simulated speech and audio features of the preset reference speech;

[0020] The loss value between the audio features of the simulated speech and the audio features of the preset reference speech is calculated through the loss function, and the autoregressive model is trained based on the loss value.

[0021] Furthermore, when the loss value satisfies a preset condition, obtaining the trained autoregressive model comprises:

[0022] When the loss value is less than a preset value, a stop instruction is generated;

[0023] The stop instruction stops training the autoregressive model, and a preset file is used to save the trained autoregressive model.

[0024] Further, the obtaining of the current text and the current reference speech, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and processing the fourth feature vector to generate a cloned speech corresponding to the current text, including:

[0025] By using the trained autoregressive model, in a self-supervised learning manner, performing self-supervised feature extraction on the current reference speech to obtain a fourth feature vector, performing style feature extraction on the current reference speech to obtain a fifth feature vector, and performing text feature extraction on the current text to obtain a sixth feature vector;

[0026] The fourth eigenvector, the fifth eigenvector, and the sixth eigenvector are processed by the residual decoder in the trained autoregressive model to generate cloned speech corresponding to the current text.

[0027] Further, after obtaining the current text and the current reference speech, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and processing the fourth feature vector to generate a cloned speech corresponding to the current text, the speech cloning method includes:

[0028] A display page corresponding to the current text is obtained, a play window of the display page is created, and the cloned voice is played through the play window.

[0029] In a second aspect, a voice cloning device based on an autoregressive model is provided, comprising:

[0030] A first acquisition module, used to acquire an original text and a preset reference voice corresponding to the original text;

[0031] An extraction module, configured to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, perform style feature extraction on the preset reference speech to obtain a second feature vector, and perform text feature extraction on the original text to obtain a third feature vector;

[0032] A determination module, configured to determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector;

[0033] A second acquisition module, used for acquiring a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value;

[0034] A third acquisition module is used to acquire the trained autoregressive model when the loss value meets a preset condition;

[0035] A generation module is used to obtain the current text and the current reference speech, perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, process the fourth feature vector, and generate a cloned speech corresponding to the current text.

[0036] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned voice cloning method when executing the computer program.

[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned voice cloning method are implemented.

[0038] The present application provides a voice cloning method, apparatus, computer equipment and storage medium based on an autoregressive model, which obtain an original text and a preset reference voice corresponding to the original text; perform self-supervised feature extraction on the preset reference voice in a self-supervised learning manner through the autoregressive model to obtain a first feature vector, perform style feature extraction on the preset reference voice to obtain a second feature vector, and perform text feature extraction on the original text to obtain a third feature vector; determine a simulated voice corresponding to the original text based on the first feature vector, the second feature vector and the third feature vector; obtain a loss value between the simulated voice and the preset reference voice, and train the autoregressive model based on the loss value; when the loss value meets a preset condition, obtain the trained autoregressive model; obtain a current text and a current reference voice, and use the trained autoregressive model to In a self-supervised learning manner, self-supervised feature extraction is performed on the current reference speech to obtain a fourth feature vector, the fourth feature vector is processed, and a cloned speech corresponding to the current text is generated. The beneficial effects are in two aspects. On the one hand, the current text and the current reference speech are obtained, and the trained autoregressive model is used to perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner to obtain a fourth feature vector, the fourth feature vector is processed, and a cloned speech corresponding to the current text is generated. Since the current reference speech is subjected to self-supervised feature extraction, the self-supervised features of the current reference speech are learned, and therefore, the cloned speech corresponding to the current text is personalized and is conducive to meeting application requirements. On the other hand, the trained autoregressive model will not be affected by human intervention, and is therefore conducive to improving the reliability of the cloned speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0040] Figure 1 is a schematic diagram of an application environment of a voice cloning method in one embodiment of the present invention;

[0041] Figure 2 A schematic flow chart of a voice cloning method provided by an embodiment of the present invention;

[0042] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S23;

[0043] Figure 4yes Figure 2 A schematic flow chart of a specific implementation of step S25;

[0044] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S26;

[0045] Figure 6 is a structural schematic diagram of a voice cloning device in one embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0047] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a voice cloning method according to an embodiment of the present invention. The voice cloning method provided by the embodiment of the present invention can be applied in the following embodiments: Figure 1 In an application environment, a client communicates with a server through a network.

[0050] The server obtains the original text and the preset reference voice corresponding to the original text through the client;

[0051] By using an autoregressive model, in a self-supervised learning manner, self-supervised feature extraction is performed on the preset reference speech to obtain a first feature vector, style feature extraction is performed on the preset reference speech to obtain a second feature vector, and text feature extraction is performed on the original text to obtain a third feature vector;

[0052] Determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector;

[0053] Obtaining a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value;

[0054] When the loss value meets the preset condition, obtaining the trained autoregressive model;

[0055] The current text and the current reference speech are obtained, and self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text.

[0056] In the scheme implemented by the above-mentioned voice cloning method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, the current text and the current reference voice are obtained, and the self-supervised feature extraction is performed on the current reference voice in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned voice corresponding to the current text. Since the self-supervised feature extraction is performed on the current reference voice, the self-supervised features of the current reference voice are learned. Therefore, the generated cloned voice corresponding to the current text is personalized and is conducive to meeting application requirements. On the other hand, the trained autoregressive model will not be affected by human intervention, so it is conducive to improving the reliability of the cloned voice.

[0057] The device running the client is referred to as a client device.

[0058] The device running the server is referred to as the server device.

[0059] The client devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0060] The server device can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0061] See also Figure 2 , Figure 2 A flow chart of a voice cloning method provided by an embodiment of the present invention includes the following steps:

[0062] S21, obtaining an original text and a preset reference voice corresponding to the original text;

[0063] The preset reference voice is the voice of a preset speaker.

[0064] For example, by recording the sound of a preset speaker reading an original text through a recording device, a preset reference voice is obtained.

[0065] S22, performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, performing style feature extraction on the preset reference speech to obtain a second feature vector, and performing text feature extraction on the original text to obtain a third feature vector;

[0066] The method of performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, performing style feature extraction on the preset reference speech to obtain a second feature vector, and performing text feature extraction on the original text to obtain a third feature vector includes:

[0067] Performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through a residual encoder in an autoregressive model to obtain a first feature vector;

[0068] Extracting style features of the preset reference speech through a style encoder in an autoregressive model to obtain a second feature vector;

[0069] The text features of the original text are extracted through the prior encoder in the autoregressive model to obtain the third feature vector.

[0070] Among them, the autoregressive model is an important statistical and machine learning model that models the intrinsic dependencies of sequence data.

[0071] Among them, the preset reference speech is subjected to self-supervised feature extraction by the residual encoder in the autoregressive model in a self-supervised learning manner to obtain a first feature vector. The first feature vector can accurately reflect the voice uniqueness of the preset reference speech, such as the timbre, intonation, pronunciation style and other personalized information of the preset reference speech, by capturing the self-supervised features in the speech signal, which is beneficial to the autoregressive model in learning the personalized information of the preset reference speech.

[0072] S23, determining the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector;

[0073] The simulated speech corresponding to the original text is the simulated speech converted from the original text. As a direct reflection of the output of the autoregressive model, the simulated speech can reflect the current performance and training progress of the autoregressive model in real time.

[0074] S24, obtaining a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value;

[0075] The step of obtaining a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value, includes:

[0076] Processing the simulated speech and the preset reference speech through a posterior encoder in an autoregressive model to obtain audio features of the simulated speech and audio features of the preset reference speech;

[0077] The loss value between the audio features of the simulated speech and the audio features of the preset reference speech is calculated through the loss function, and the autoregressive model is trained based on the loss value.

[0078] The autoregressive model is trained with the goal of minimizing the loss value. The smaller the loss value, the smaller the error between the audio features of the simulated speech and the audio features of the preset reference speech, and the stronger the performance of the autoregressive model. Therefore, by continuously adjusting the parameters of the autoregressive model, the loss value is gradually reduced, thereby improving the performance of the autoregressive model.

[0079] The loss function includes a mean square error loss function and a cross entropy loss function, or a combination thereof.

[0080] S25, when the loss value meets the preset condition, obtaining the trained autoregressive model;

[0081] Among them, the trained model can perform efficient speech synthesis based on the learned rules and features, thereby greatly improving the speed and accuracy of speech synthesis.

[0082] S26, obtaining the current text and the current reference speech, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, processing the fourth feature vector to generate a cloned speech corresponding to the current text.

[0083] The current reference speech is the voice of the target speaker.

[0084] The target speaker refers to the individual or entity whose voice characteristics are copied through cloning technology.

[0085] For example, the target speaker is the customer service of the target platform.

[0086] Target platforms include financial platforms, technology platforms, and insurance platforms.

[0087] Customer service plays a vital role in the target platform. Customer service interacts with customers through voice, answers questions, provides help, and maintains customer satisfaction. Each customer service may have its own unique voice characteristics, tone, speed, and speaking style, which together constitute the voice image of the customer service.

[0088] By cloning the customer service's voice, customers can be provided with a more personalized service experience. For example, when a customer interacts with an AI assistant, the AI ​​assistant can simulate the customer service's voice, making the customer feel like they are communicating with a real person.

[0089] Among them, artificial intelligence assistant refers to an intelligent system that uses artificial intelligence technology, especially natural language processing, machine learning and other technologies, to provide assistance, support or suggestions to users.

[0090] The cloned voice corresponding to the current text is the cloned voice converted from the current text. Since the cloned voice is highly similar to the current reference voice, a personalized cloned voice is generated, which can bring a more real and natural auditory experience to the user and is conducive to meeting application requirements.

[0091] The trained autoregressive model is used to extract self-supervised features of the current reference speech in a self-supervised learning manner to obtain a fourth eigenvector. The fourth eigenvector can accurately reflect the voice uniqueness of the current reference speech, such as the timbre, intonation, pronunciation style and other personalized information of the current reference speech, by capturing the self-supervised features in the speech signal, which is beneficial for the autoregressive model to learn the personalized information of the current reference speech.

[0092] Wherein, after obtaining the current text and the current reference speech, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and processing the fourth feature vector to generate a cloned speech corresponding to the current text, the speech cloning method includes:

[0093] A display page corresponding to the current text is obtained, a play window of the display page is created, and the cloned voice is played through the play window.

[0094] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, the current text and the current reference speech are obtained, and the self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text. Since the self-supervised feature extraction is performed on the current reference speech, the self-supervised features of the current reference speech are learned. Therefore, the cloned speech corresponding to the current text is personalized and is conducive to meeting application requirements. On the other hand, the trained autoregressive model will not be affected by human intervention, so it is conducive to improving the reliability of the cloned speech.

[0095] See also Figure 3 , Figure 3 yes Figure 2 A specific implementation flow diagram of step S23 is described in detail as follows:

[0096] S31, inputting the first feature vector, the second feature vector, and the third feature vector into a residual decoder in an autoregressive model;

[0097] S32, processing the first feature vector, the second feature vector, and the third feature vector through a residual decoder in an autoregressive model to generate simulated speech corresponding to the original text.

[0098] The first feature vector, the second feature vector, and the third feature vector are processed by a residual decoder in an autoregressive model to generate simulated speech corresponding to the original text, including:

[0099] The first feature vector, the second feature vector, and the third feature vector are integrated to obtain a first target vector, and the first target vector is processed by a residual decoder in an autoregressive model to generate simulated speech corresponding to the original text.

[0100] In an embodiment of the present invention, simulated speech corresponding to the original text is generated. This diversity is conducive to the autoregressive model learning a wider range of speech features, thereby improving the generalization ability of the autoregressive model.

[0101] See also Figure 4 , Figure 4 yes Figure 2 A specific implementation flow diagram of step S25 is described in detail as follows:

[0102] S41, when the loss value is less than a preset value, generating a stop instruction;

[0103] S42, stopping the training of the autoregressive model through the stop instruction, and using a preset file to save the trained autoregressive model.

[0104] In an embodiment of the present invention, a preset file is used to save the trained autoregressive model. The trained autoregressive model can be loaded at any time without repeating the entire time-consuming training process, which greatly improves the use efficiency of the trained autoregressive model.

[0105] In summary, saving the trained model not only helps improve work efficiency, but also promotes team collaboration and knowledge inheritance, and provides strong protection for data security.

[0106] See also Figure 5 , Figure 5 yes Figure 2 A specific implementation flow diagram of step S26 is described in detail as follows:

[0107] S51, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, performing style feature extraction on the current reference speech to obtain a fifth feature vector, and performing text feature extraction on the current text to obtain a sixth feature vector;

[0108] Among them, the self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, the style feature extraction is performed on the current reference speech to obtain a fifth feature vector, and the text feature extraction is performed on the current text to obtain a sixth feature vector, including:

[0109] Performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner by using the residual encoder in the trained autoregressive model to obtain a fourth feature vector;

[0110] Extracting style features of the current reference speech by using a style encoder in the trained autoregressive model to obtain a fifth feature vector;

[0111] The text feature extraction is performed on the current text through the prior encoder in the trained autoregressive model to obtain the sixth feature vector.

[0112] S52, processing the fourth feature vector, the fifth feature vector, and the sixth feature vector through the trained residual decoder in the autoregressive model to generate a cloned speech corresponding to the current text.

[0113] The fourth feature vector, the fifth feature vector, and the sixth feature vector are processed by the residual decoder in the trained autoregressive model to generate a cloned speech corresponding to the current text, including:

[0114] The fourth eigenvector, the fifth eigenvector, and the sixth eigenvector are integrated to obtain a second target vector, and the second target vector is processed by the residual decoder in the trained autoregressive model to generate a cloned speech corresponding to the current text.

[0115] In the embodiment of the present invention, the fourth eigenvector, the fifth eigenvector, and the sixth eigenvector are processed by the residual decoder in the trained autoregressive model to generate a cloned speech corresponding to the current text. Since the self-supervisory feature extraction is performed on the current reference speech, the self-supervisory features of the current reference speech are learned. Therefore, the cloned speech corresponding to the current text is personalized and is conducive to meeting application requirements.

[0116] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a voice cloning device according to an embodiment of the present invention. Figure 6 As shown, the voice cloning device includes a first acquisition module 101, an extraction module 102, a determination module 103, a second acquisition module 104, a third acquisition module 105, and a generation module 106. The functional modules are described in detail as follows:

[0117] A first acquisition module 101 is used to acquire an original text and a preset reference speech corresponding to the original text;

[0118] The extraction module 102 is used to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, perform style feature extraction on the preset reference speech to obtain a second feature vector, and perform text feature extraction on the original text to obtain a third feature vector;

[0119] A determination module 103, configured to determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector;

[0120] A second acquisition module 104 is used to acquire a loss value between the simulated speech and the preset reference speech, and train the autoregressive model based on the loss value;

[0121] A third acquisition module 105 is used to acquire the trained autoregressive model when the loss value meets a preset condition;

[0122] The generation module 106 is used to obtain the current text and the current reference speech, perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, process the fourth feature vector, and generate a cloned speech corresponding to the current text.

[0123] In one embodiment, the extraction module 102 includes:

[0124] A first extraction subunit is used to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through a residual encoder in an autoregressive model to obtain a first feature vector;

[0125] A second extraction subunit is used to extract style features of the preset reference speech through a style encoder in an autoregressive model to obtain a second feature vector;

[0126] The third extraction subunit is used to extract text features from the original text through a priori encoder in the autoregressive model to obtain a third feature vector.

[0127] In one embodiment, the determining module 103 includes:

[0128] An input subunit, used for inputting the first feature vector, the second feature vector, and the third feature vector into a residual decoder in an autoregressive model;

[0129] The first processing subunit is used to process the first feature vector, the second feature vector, and the third feature vector through a residual decoder in an autoregressive model to generate simulated speech corresponding to the original text.

[0130] In one embodiment, the second acquisition module 104 includes:

[0131] A second processing subunit is used to process the simulated speech and the preset reference speech through a posterior encoder in an autoregressive model to obtain audio features of the simulated speech and audio features of the preset reference speech;

[0132] The calculation subunit is used to calculate the loss value between the audio features of the simulated speech and the audio features of the preset reference speech through a loss function, and train the autoregressive model based on the loss value.

[0133] In one embodiment, the third acquisition module 105 includes:

[0134] A first generating subunit, configured to generate a stop instruction when the loss value is less than a preset value;

[0135] The saving subunit is used to stop training the autoregressive model through the stop instruction, and use a preset file to save the trained autoregressive model.

[0136] In one embodiment, the generating module 106 includes:

[0137] The fourth extraction subunit is used to perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, perform style feature extraction on the current reference speech to obtain a fifth feature vector, and perform text feature extraction on the current text to obtain a sixth feature vector;

[0138] The second generating subunit is used to process the fourth feature vector, the fifth feature vector, and the sixth feature vector through the trained residual decoder in the autoregressive model to generate the cloned speech corresponding to the current text.

[0139] In one embodiment, the voice cloning device further includes:

[0140] The display module is used to obtain a display page corresponding to the current text, create a play window of the display page, and play the cloned voice through the play window.

[0141] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, the current text and the current reference speech are obtained, and the self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text. Since the self-supervised feature extraction is performed on the current reference speech, the self-supervised features of the current reference speech are learned. Therefore, the cloned speech corresponding to the current text is personalized and is conducive to meeting application requirements. On the other hand, the trained autoregressive model will not be affected by human intervention, so it is conducive to improving the reliability of the cloned speech.

[0142] For the specific definition of the voice cloning device, please refer to the definition of the voice cloning method above, which will not be repeated here.

[0143] Each module in the above-mentioned voice cloning device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0144] See also Figure 7 , Figure 7is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.

[0145] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the functions or steps of a voice cloning method based on an autoregressive model can be implemented on the server device.

[0146] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the functions or steps of a voice cloning method based on an autoregressive model can be implemented on a client device.

[0147] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.

[0148] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0151] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the non-Company software tools or components that appear in the embodiments of the present application are only examples and do not represent actual use.

Claims

1. A voice cloning method based on an autoregressive model, characterized in that: include: Acquire an original text and a preset reference voice corresponding to the original text; By using an autoregressive model, in a self-supervised learning manner, self-supervised feature extraction is performed on the preset reference speech to obtain a first feature vector, style feature extraction is performed on the preset reference speech to obtain a second feature vector, and text feature extraction is performed on the original text to obtain a third feature vector; Determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector; Obtaining a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value; When the loss value meets the preset condition, obtaining the trained autoregressive model; The current text and the current reference speech are obtained, and self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text.

2. The voice cloning method according to claim 1, characterized in that: The method of performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, performing style feature extraction on the preset reference speech to obtain a second feature vector, and performing text feature extraction on the original text to obtain a third feature vector includes: Performing self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through a residual encoder in an autoregressive model to obtain a first feature vector; The style feature of the preset reference speech is extracted by the style encoder in the autoregressive model to obtain a second feature vector; and the text feature of the original text is extracted by the priori encoder in the autoregressive model to obtain a third feature vector.

3. The voice cloning method according to claim 1, characterized in that: The step of determining the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector includes: Input the first feature vector, the second feature vector, and the third feature vector into a residual decoder in an autoregressive model; The first feature vector, the second feature vector, and the third feature vector are processed by a residual decoder in an autoregressive model to generate simulated speech corresponding to the original text.

4. The voice cloning method according to claim 1, characterized in that: The obtaining of the loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value, comprises: Processing the simulated speech and the preset reference speech through a posterior encoder in an autoregressive model to obtain audio features of the simulated speech and audio features of the preset reference speech; The loss value between the audio features of the simulated speech and the audio features of the preset reference speech is calculated through the loss function, and the autoregressive model is trained based on the loss value.

5. The voice cloning method according to claim 1, characterized in that: When the loss value satisfies a preset condition, obtaining the trained autoregressive model comprises: When the loss value is less than a preset value, a stop instruction is generated; The stop instruction stops training the autoregressive model, and a preset file is used to save the trained autoregressive model.

6. The voice cloning method according to claim 1, characterized in that: The step of obtaining the current text and the current reference speech, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and processing the fourth feature vector to generate a cloned speech corresponding to the current text includes: By using the trained autoregressive model, in a self-supervised learning manner, performing self-supervised feature extraction on the current reference speech to obtain a fourth feature vector, performing style feature extraction on the current reference speech to obtain a fifth feature vector, and performing text feature extraction on the current text to obtain a sixth feature vector; The fourth eigenvector, the fifth eigenvector, and the sixth eigenvector are processed by the residual decoder in the trained autoregressive model to generate cloned speech corresponding to the current text.

7. The voice cloning method according to any one of claims 1 to 6, characterized in that: After obtaining the current text and the current reference speech, performing self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and processing the fourth feature vector to generate a cloned speech corresponding to the current text, the speech cloning method includes: A display page corresponding to the current text is obtained, a play window of the display page is created, and the cloned voice is played through the play window.

8. A voice cloning device based on an autoregressive model, characterized in that: include: A first acquisition module, used to acquire an original text and a preset reference voice corresponding to the original text; An extraction module, configured to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, perform style feature extraction on the preset reference speech to obtain a second feature vector, and perform text feature extraction on the original text to obtain a third feature vector; A determination module, configured to determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector; A second acquisition module, used for acquiring a loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value; A third acquisition module is used to acquire the trained autoregressive model when the loss value meets a preset condition; A generation module is used to obtain the current text and the current reference speech, perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, process the fourth feature vector, and generate a cloned speech corresponding to the current text.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the voice cloning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the voice cloning method according to any one of claims 1 to 7 are implemented.

Images