Speech cloning method and device based on autoregressive model, equipment and storage medium

Abstract

The present application relates to the technical field of artificial intelligence, and can be applied to the field of financial technology, and discloses a voice cloning method and device based on an autoregressive model, equipment and a storage medium, the method comprising: performing self-supervised feature extraction on a preset reference voice to obtain a first feature vector, performing style feature extraction on the preset reference voice to obtain a second feature vector, and performing text feature extraction on an original text to obtain a third feature vector; determining simulated voice corresponding to the original text based on the first feature vector, the second feature vector and the third feature vector; obtaining a loss value between the simulated voice and the preset reference voice, training an autoregressive model based on the loss value; and performing self-supervised feature extraction on a current reference voice through the trained autoregressive model to obtain a fourth feature vector, processing the fourth feature vector, and generating cloned voice corresponding to the current text. The present application can generate personalized cloned voice, which is conducive to meeting application requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and can be applied to the field of financial technology, and in particular relates to a voice cloning method and device based on an autoregressive model, equipment and a storage medium. BACKGROUND

[0002] With the continuous progress of artificial intelligence technology, voice cloning technology has ushered in unprecedented development opportunities and has shown broad application potential in many fields.

[0003] Due to the limitations of the learning mechanism, the voice cloning technology is difficult to capture self-supervised features, which are useful information automatically extracted from a large amount of unlabeled data. Due to the inability to learn self-supervised features, the cloned voice generated by the voice cloning technology lacks personalization, which limits the application scenarios of the voice cloning technology. Therefore, the cloned voice generated by the existing voice cloning method lacks personalization, which is not conducive to meeting application requirements. SUMMARY

[0004] The present application provides a voice cloning method and device based on an autoregressive model, a computer device and a storage medium to solve the technical problem of the cloned voice generated by the existing voice cloning method lacking personalization.

[0005] In a first aspect, a voice cloning method based on an autoregressive model is provided, comprising:

[0006] obtaining an original text and a preset reference voice corresponding to the original text;

[0007] extracting self-supervised features from the preset reference voice in a self-supervised learning manner by an autoregressive model to obtain a first feature vector, extracting style features from the preset reference voice to obtain a second feature vector, and extracting text features from the original text to obtain a third feature vector;

[0008] determining an analog voice corresponding to the original text based on the first feature vector, the second feature vector and the third feature vector;

[0009] obtaining a loss value between the analog voice and the preset reference voice, and training the autoregressive model based on the loss value;

[0010] obtaining the trained autoregressive model when the loss value meets a preset condition;

[0011] Obtaining the current text and the current reference voice, performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, processing the fourth feature vector to generate cloned voice corresponding to the current text.

[0012] Further, the self-supervised feature extraction on the preset reference voice in a self-supervised learning manner through the autoregressive model to obtain a first feature vector, the style feature extraction on the preset reference voice to obtain a second feature vector, and the text feature extraction on the original text to obtain a third feature vector include:

[0013] The self-supervised feature extraction on the preset reference voice in a self-supervised learning manner through the residual encoder in the autoregressive model to obtain a first feature vector;

[0014] The style feature extraction on the preset reference voice through the style encoder in the autoregressive model to obtain a second feature vector, and the text feature extraction on the original text through the prior encoder in the autoregressive model to obtain a third feature vector.

[0015] Further, the determination of the simulated voice corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector includes:

[0016] The first feature vector, the second feature vector, and the third feature vector are input into a residual decoder in the autoregressive model.

[0017] The first feature vector, the second feature vector, and the third feature vector are processed through the residual decoder in the autoregressive model to generate the simulated voice corresponding to the original text.

[0018] Further, the obtaining of the loss value between the simulated voice and the preset reference voice and the training of the autoregressive model based on the loss value include:

[0019] The simulated voice and the preset reference voice are processed through the posterior encoder in the autoregressive model to obtain audio features of the simulated voice and audio features of the preset reference voice.

[0020] The loss value between the audio features of the simulated voice and the audio features of the preset reference voice is calculated through a loss function, and the autoregressive model is trained based on the loss value.

[0021] Further, the obtaining of the trained autoregressive model when the loss value meets a preset condition includes:

[0022] generate a stop instruction when the loss value is less than a preset value;

[0023] stop training the autoregressive model by the stop instruction, save the trained autoregressive model by using a preset file.

[0024] Further, the current text and the current reference speech are obtained, and the current reference speech is subjected to self-supervised feature extraction in a self-supervised learning manner by the trained autoregressive model to obtain a fourth feature vector. The fourth feature vector is processed to generate a cloned speech corresponding to the current text, including:

[0025] The current reference speech is subjected to self-supervised feature extraction in a self-supervised learning manner by the trained autoregressive model to obtain a fourth feature vector. The current reference speech is subjected to style feature extraction to obtain a fifth feature vector. The current text is subjected to text feature extraction to obtain a sixth feature vector.

[0026] The fourth feature vector, the fifth feature vector, and the sixth feature vector are processed by a residual decoder in the trained autoregressive model to generate the cloned speech corresponding to the current text.

[0027] Further, after the current text and the current reference speech are obtained, and the current reference speech is subjected to self-supervised feature extraction in a self-supervised learning manner by the trained autoregressive model to obtain a fourth feature vector, the fourth feature vector is processed to generate a cloned speech corresponding to the current text, the speech cloning method includes:

[0028] A display page corresponding to the current text is obtained, a playback window of the display page is created, and the cloned speech is played through the playback window.

[0029] In a second aspect, a speech cloning device based on an autoregressive model is provided, including:

[0030] A first obtaining module is configured to obtain an original text and a preset reference speech corresponding to the original text.

[0031] An extracting module is configured to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner by an autoregressive model to obtain a first feature vector. Style feature extraction is performed on the preset reference speech to obtain a second feature vector. Text feature extraction is performed on the original text to obtain a third feature vector.

[0032] A determining module is configured to determine an analog speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector.

[0033] a second obtaining module, configured to obtain a loss value between the analog voice and the preset reference voice, and train the autoregressive model based on the loss value;

[0034] a third obtaining module, configured to obtain the trained autoregressive model when the loss value meets a preset condition;

[0035] a generating module, configured to obtain a current text and a current reference voice, perform self-supervised feature extraction on the current reference voice in a self-supervised learning manner through the trained autoregressive model, obtain a fourth feature vector, and process the fourth feature vector to generate a cloned voice corresponding to the current text.

[0036] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the voice cloning method when executing the computer program.

[0037] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the voice cloning method when executed by a processor.

[0038] The application provides a speech cloning method and device based on an autoregressive model, computer equipment and a storage medium. An original text and a preset reference speech corresponding to the original text are obtained. The preset reference speech is subjected to self-supervised feature extraction in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, the preset reference speech is subjected to style feature extraction to obtain a second feature vector, and the original text is subjected to text feature extraction to obtain a third feature vector. The first feature vector, the second feature vector and the third feature vector are used to determine an analog speech corresponding to the original text. A loss value between the analog speech and the preset reference speech is obtained, and the autoregressive model is trained based on the loss value. When the loss value meets a preset condition, the trained autoregressive model is obtained. A current text and a current reference speech are obtained, the current reference speech is subjected to self-supervised feature extraction in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text. The beneficial effects are as follows. On the one hand, the current text and the current reference speech are obtained, the current reference speech is subjected to self-supervised feature extraction in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate a cloned speech corresponding to the current text. Since the current reference speech is subjected to self-supervised feature extraction, the self-supervised features of the current reference speech are learned, thus the cloned speech corresponding to the current text has individuality, which is beneficial to meet application requirements. On the other hand, the trained autoregressive model is not affected by artificial intervention, thus the reliability of the cloned speech is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0040] Figure 1 is an application environment schematic diagram of a speech cloning method in an embodiment of the present application;

[0041] Figure 2 is a flow schematic diagram of a speech cloning method provided by an embodiment of the present application;

[0042] Figure 3 is Figure 2 is a specific implementation flow schematic diagram of step S23 in the embodiment;

[0043] Figure 4is Figure 2 a specific implementation flowchart of step S25 in the embodiment;

[0044] Figure 5 is Figure 2 a specific implementation flowchart of step S26 in the embodiment;

[0045] Figure 6 is a structural schematic diagram of a voice cloning device in an embodiment of the present application;

[0046] Figure 7 is a structural schematic diagram of a computer device in an embodiment of the present application;

[0047] Figure 8 is another structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0049] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment of a voice cloning method in an embodiment of the present application. The voice cloning method provided in the embodiments of the present application can be applied in an application environment as shown in Figure 1 , wherein a client communicates with a server through a network.

[0050] The server obtains an original text and a preset reference voice corresponding to the original text through the client.

[0051] The preset reference voice is subjected to self-supervised feature extraction through a self-recurrent model in a self-supervised learning manner to obtain a first feature vector, is subjected to style feature extraction to obtain a second feature vector, and the original text is subjected to text feature extraction to obtain a third feature vector.

[0052] Based on the first feature vector, the second feature vector and the third feature vector, a simulated voice corresponding to the original text is determined.

[0053] A loss value between the simulated voice and the preset reference voice is obtained, and the self-recurrent model is trained based on the loss value.

[0054] When the loss value meets a preset condition, the trained self-recurrent model is obtained.

[0055] obtaining a current text and a current reference voice, performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner through the trained self-recurrent model to obtain a fourth feature vector, processing the fourth feature vector, and generating cloned voice corresponding to the current text.

[0056] In the scheme implemented by the voice cloning method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, the current text and the current reference voice are obtained, the self-supervised feature extraction is performed on the current reference voice in a self-supervised learning manner through the trained self-recurrent model to obtain a fourth feature vector, the fourth feature vector is processed, and the cloned voice corresponding to the current text is generated. Since the self-supervised feature extraction is performed on the current reference voice, the self-supervised feature of the current reference voice is learned, therefore, the cloned voice corresponding to the current text is generated with personalization, which is beneficial to meet the application requirements. On the other hand, the trained self-recurrent model will not be affected by human intervention, therefore, it is beneficial to improve the reliability of the cloned voice.

[0057] Among them, the device running the client is referred to as: client device.

[0058] Among them, the device running the server is referred to as: server device.

[0059] Among them, the client device can include but is not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices.

[0060] Among them, the server device can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0061] Please refer to Figure 2 , Figure 2 A flowchart of a voice cloning method provided by an embodiment of the present application includes the following steps:

[0062] S21, obtaining an original text and a preset reference voice corresponding to the original text;

[0063] Among them, the preset reference voice is the voice of a preset speaker.

[0064] For example, the voice of the preset speaker reading the original text is recorded by a recording device to obtain the preset reference voice.

[0065] S22, performing self-supervised feature extraction on the preset reference voice by using an autoregressive model in a self-supervised learning manner to obtain a first feature vector, performing style feature extraction on the preset reference voice to obtain a second feature vector, and performing text feature extraction on the original text to obtain a third feature vector;

[0066] The self-supervised feature extraction on the preset reference voice by using the autoregressive model in the self-supervised learning manner to obtain the first feature vector, the style feature extraction on the preset reference voice to obtain the second feature vector, and the text feature extraction on the original text to obtain the third feature vector include:

[0067] The self-supervised feature extraction on the preset reference voice by using the residual encoder in the autoregressive model in the self-supervised learning manner to obtain the first feature vector;

[0068] The style feature extraction on the preset reference voice by using the style encoder in the autoregressive model to obtain the second feature vector;

[0069] The text feature extraction on the original text by using the prior encoder in the autoregressive model to obtain the third feature vector.

[0070] The autoregressive model is an important statistical and machine learning model, which is based on the internal dependence of sequence data for modeling.

[0071] The self-supervised feature extraction on the preset reference voice by using the residual encoder in the autoregressive model in the self-supervised learning manner to obtain the first feature vector. The first feature vector can accurately reflect the unique characteristics of the preset reference voice, such as the timbre, intonation, pronunciation style and other personalized information of the preset reference voice, by capturing the self-supervised features in the voice signal, which is conducive to the autoregressive model to learn the personalized information of the preset reference voice.

[0072] S23, determining the simulated voice corresponding to the original text based on the first feature vector, the second feature vector and the third feature vector;

[0073] The simulated voice corresponding to the original text is the simulated voice converted from the original text. The simulated voice as the direct embodiment of the output of the autoregressive model can reflect the current performance and training progress of the autoregressive model in real time.

[0074] S24, obtaining a loss value between the simulated voice and the preset reference voice, and training the autoregressive model based on the loss value;

[0075] The loss value between the simulated voice and the preset reference voice is obtained, and the autoregressive model is trained based on the loss value, including:

[0076] The simulated voice and the preset reference voice are processed through a posterior encoder in the autoregressive model to obtain audio features of the simulated voice and audio features of the preset reference voice.

[0077] A loss value between the audio features of the simulated voice and the audio features of the preset reference voice is calculated through a loss function, and the autoregressive model is trained based on the loss value.

[0078] The autoregressive model is trained with the goal of minimizing the loss value. The smaller the loss value, the smaller the error between the audio features of the simulated voice and the audio features of the preset reference voice, and the stronger the performance of the autoregressive model. Therefore, by continuously adjusting the parameters of the autoregressive model, the loss value is gradually reduced, and the performance of the autoregressive model is improved.

[0079] The loss function includes one or a combination of a mean square error loss function and a cross-entropy loss function.

[0080] S25, when the loss value meets a preset condition, the trained autoregressive model is obtained.

[0081] The trained model can perform efficient speech synthesis based on learned rules and features, thereby greatly improving the speed and accuracy of speech synthesis.

[0082] S26, obtain a current text and a current reference voice, and perform self-supervised feature extraction on the current reference voice in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and process the fourth feature vector to generate a cloned voice corresponding to the current text.

[0083] The current reference voice is the voice of a target speaker.

[0084] The target speaker refers to an individual or entity whose voice features are copied through cloning technology.

[0085] For example, the target speaker is a customer service of a target platform.

[0086] The target platform includes a financial platform, a technology platform, and an insurance platform.

[0087] Customer service plays a crucial role in the target platform, and customer service interacts with customers through voice, answers questions, provides help, and maintains customer satisfaction. Each customer service may have its own unique voice characteristics, tone, speed, and speaking style, which together constitute the voice image of the customer service.

[0088] By cloning the voice of the customer service, a more personalized service experience can be provided to the customer. For example, when a customer interacts with an artificial intelligence assistant, the artificial intelligence assistant can simulate the voice of the customer service, making the customer feel like they are communicating with a real person.

[0089] Among them, the artificial intelligence assistant refers to an intelligent system that uses artificial intelligence technology, especially natural language processing, machine learning, and other technologies, to provide assistance, support, or suggestions to users.

[0090] Among them, the cloned voice corresponding to the current text is the cloned voice converted from the current text. Since the cloned voice is highly similar to the current reference voice, the personalized cloned voice is generated, which can bring users a more real and natural auditory experience, and is conducive to meeting application requirements.

[0091] Among them, the fourth feature vector is obtained by performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner through the trained autoregressive model. The fourth feature vector can accurately reflect the unique characteristics of the current reference voice by capturing self-supervised features in the voice signal, such as the timbre, tone, pronunciation style, and other personalized information of the current reference voice, which is conducive to the autoregressive model learning personalized information of the current reference voice.

[0092] Among them, after obtaining the current text and the current reference voice, performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner through the trained autoregressive model to obtain a fourth feature vector, and processing the fourth feature vector to generate the cloned voice corresponding to the current text, the voice cloning method comprises:

[0093] Obtaining a display page corresponding to the current text, creating a playback window of the display page, and playing the cloned voice through the playback window.

[0094] In the embodiment of the present application, the beneficial effects are in two aspects. On the one hand, the current text and the current reference voice are obtained, the self-supervised feature extraction is performed on the current reference voice in a self-supervised learning manner through the trained self-recurrent model to obtain a fourth feature vector, and the fourth feature vector is processed to generate the cloned voice corresponding to the current text. Since the self-supervised feature extraction is performed on the current reference voice, the self-supervised feature of the current reference voice is learned, thus the cloned voice corresponding to the current text is personalized, which is beneficial to meet the application requirements. On the other hand, the trained self-recurrent model is not affected by human intervention, thus the reliability of the cloned voice is improved.

[0095] Please refer to Figure 3 , Figure 3 is Figure 2 a specific implementation flowchart of step S23 in

[0096] S31, input the first feature vector, the second feature vector and the third feature vector into a residual decoder in a self-recurrent model.

[0097] S32, process the first feature vector, the second feature vector and the third feature vector through the residual decoder in the self-recurrent model to generate the simulated voice corresponding to the original text.

[0098] Wherein, processing the first feature vector, the second feature vector and the third feature vector through the residual decoder in the self-recurrent model to generate the simulated voice corresponding to the original text comprises:

[0099] integrating the first feature vector, the second feature vector and the third feature vector to obtain a first target vector, and processing the first target vector through the residual decoder in the self-recurrent model to generate the simulated voice corresponding to the original text.

[0100] In the embodiment of the present application, the simulated voice corresponding to the original text is generated, which is beneficial to the self-recurrent model to learn more extensive voice features, thereby improving the generalization ability of the self-recurrent model.

[0101] Please refer to Figure 4 , Figure 4 is Figure 2 a specific implementation flowchart of step S25 in

[0102] S41, when the loss value is less than a preset value, a stop instruction is generated.

[0103] S42, stop training the autoregressive model by the stop instruction, save the trained autoregressive model by using a preset file.

[0104] In the embodiment of the present application, the trained autoregressive model is saved by using a preset file, and the trained autoregressive model is loaded at any time without repeating the entire time-consuming training process, thereby greatly improving the use efficiency of the trained autoregressive model.

[0105] In summary, saving the trained model not only helps to improve work efficiency, but also promotes team collaboration and knowledge inheritance, and provides strong protection for data security.

[0106] Please refer to Figure 5 , Figure 5 is Figure 2 a specific implementation flowchart of step S26, which is described in detail as follows:

[0107] S51, by the trained autoregressive model, performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner to obtain a fourth feature vector, performing style feature extraction on the current reference voice to obtain a fifth feature vector, and performing text feature extraction on the current text to obtain a sixth feature vector;

[0108] Among them, by the trained autoregressive model, performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner to obtain a fourth feature vector, performing style feature extraction on the current reference voice to obtain a fifth feature vector, and performing text feature extraction on the current text to obtain a sixth feature vector, including:

[0109] By the residual encoder in the trained autoregressive model, performing self-supervised feature extraction on the current reference voice in a self-supervised learning manner to obtain a fourth feature vector;

[0110] By the style encoder in the trained autoregressive model, performing style feature extraction on the current reference voice to obtain a fifth feature vector;

[0111] By the prior encoder in the trained autoregressive model, performing text feature extraction on the current text to obtain a sixth feature vector.

[0112] S52, by the residual decoder in the trained autoregressive model, processing the fourth feature vector, the fifth feature vector and the sixth feature vector to generate a cloned voice corresponding to the current text.

[0113] The fourth feature vector, the fifth feature vector and the sixth feature vector are processed through a residual decoder in the trained autoregressive model to generate cloned speech corresponding to the current text, including:

[0114] The fourth feature vector, the fifth feature vector and the sixth feature vector are integrated to obtain a second target vector, and the second target vector is processed through a residual decoder in the trained autoregressive model to generate cloned speech corresponding to the current text.

[0115] In the embodiment of the application, the fourth feature vector, the fifth feature vector and the sixth feature vector are processed through a residual decoder in the trained autoregressive model to generate cloned speech corresponding to the current text. Since the self-supervised feature extraction is performed on the current reference speech, the self-supervised feature of the current reference speech is learned, so that the cloned speech corresponding to the current text is personalized, which is beneficial to meet the application requirements.

[0116] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of a speech cloning device in an embodiment of the application, as Figure 6 shown, the speech cloning device comprises a first acquisition module 101, an extraction module 102, a determination module 103, a second acquisition module 104, a third acquisition module 105 and a generation module 106. The detailed description of each functional module is as follows:

[0117] The first acquisition module 101 is configured to acquire an original text and a preset reference speech corresponding to the original text.

[0118] The extraction module 102 is configured to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through an autoregressive model to obtain a first feature vector, perform style feature extraction on the preset reference speech to obtain a second feature vector, and perform text feature extraction on the original text to obtain a third feature vector.

[0119] The determination module 103 is configured to determine an analog speech corresponding to the original text based on the first feature vector, the second feature vector and the third feature vector.

[0120] The second acquisition module 104 is configured to acquire a loss value between the analog speech and the preset reference speech, and train the autoregressive model based on the loss value.

[0121] The third acquisition module 105 is configured to acquire the trained autoregressive model when the loss value meets a preset condition.

[0122] The generating module 106 is configured to acquire the current text and the current reference speech, perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner through the trained self-recurrent model, obtain a fourth feature vector, and process the fourth feature vector to generate cloned speech corresponding to the current text.

[0123] In one embodiment, the extracting module 102 comprises:

[0124] The first extracting subunit is configured to perform self-supervised feature extraction on the preset reference speech in a self-supervised learning manner through a residual encoder in the self-recurrent model to obtain a first feature vector.

[0125] The second extracting subunit is configured to perform style feature extraction on the preset reference speech through a style encoder in the self-recurrent model to obtain a second feature vector.

[0126] The third extracting subunit is configured to perform text feature extraction on the original text through a priori encoder in the self-recurrent model to obtain a third feature vector.

[0127] In one embodiment, the determining module 103 comprises:

[0128] The input subunit is configured to input the first feature vector, the second feature vector, and the third feature vector into a residual decoder in the self-recurrent model.

[0129] The first processing subunit is configured to process the first feature vector, the second feature vector, and the third feature vector through the residual decoder in the self-recurrent model to generate simulated speech corresponding to the original text.

[0130] In one embodiment, the second acquiring module 104 comprises:

[0131] The second processing subunit is configured to process the simulated speech and the preset reference speech through a posteriori encoder in the self-recurrent model to obtain an audio feature of the simulated speech and an audio feature of the preset reference speech.

[0132] The calculating subunit is configured to calculate a loss value between the audio feature of the simulated speech and the audio feature of the preset reference speech through a loss function, and train the self-recurrent model based on the loss value.

[0133] In one embodiment, the third acquiring module 105 comprises:

[0134] The first generating subunit is configured to generate a stop instruction when the loss value is less than a preset value.

[0135] The saving subunit is configured to, by the stopping instruction, stop training the autoregressive model, and save the trained autoregressive model by using a preset file.

[0136] In one embodiment, the generating module 106 comprises:

[0137] The fourth extracting subunit is configured to, by the trained autoregressive model, perform self-supervised feature extraction on the current reference voice in a self-supervised learning manner to obtain a fourth feature vector, perform style feature extraction on the current reference voice to obtain a fifth feature vector, and perform text feature extraction on the current text to obtain a sixth feature vector.

[0138] The second generating subunit is configured to, by the residual decoder in the trained autoregressive model, process the fourth feature vector, the fifth feature vector, and the sixth feature vector to generate cloned voice corresponding to the current text.

[0139] In one embodiment, the voice cloning apparatus further comprises:

[0140] The display module is configured to obtain a display page corresponding to the current text, create a playing window of the display page, and play the cloned voice through the playing window.

[0141] In the embodiments of the present application, the beneficial effects are in two aspects. On the one hand, the current text and the current reference voice are obtained, the current reference voice is extracted in a self-supervised manner by the trained autoregressive model to obtain a fourth feature vector, and the fourth feature vector is processed to generate cloned voice corresponding to the current text. Since the current reference voice is extracted in a self-supervised manner, the self-supervised features of the current reference voice are learned, thus the cloned voice corresponding to the current text is personalized, which is conducive to meeting application requirements. On the other hand, the trained autoregressive model is not affected by human intervention, thus the reliability of the cloned voice is improved.

[0142] The specific limitations of the voice cloning apparatus can be referred to the limitations of the voice cloning method in the above, which will not be repeated here.

[0143] The above voice cloning apparatus can be realized by software, hardware, and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0144] Please refer to Figure 7 , Figure 7is a structural schematic diagram of a computer device in an embodiment of the present application, in an embodiment, a computer device is provided, which can be a server device, and an internal structure diagram of the computer device can be as shown in Figure 7 . The computer device includes a processor, a memory, a network interface and a database connected through a system bus.

[0145] The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and / or an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The computer program, when executed by the processor, can implement functions or steps of a speech cloning method based on an autoregressive model on the server device.

[0146] Please refer to Figure 8 , Figure 8 is another structural schematic diagram of a computer device in an embodiment of the present application, in an embodiment, a computer device is provided, which can be a client device, and an internal structure diagram of the computer device can be as shown in Figure 8 . The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The computer program, when executed by the processor, can implement functions or steps of a speech cloning method based on an autoregressive model on the client device.

[0147] In an embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor.

[0148] It should be noted that the functions or steps that can be implemented by the computer readable storage medium or the computer device described above can be referred to the related description in the foregoing method embodiments, and will not be described here again to avoid repetition.

[0149] The processor described above can be a general processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0150] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software, it can depend on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0151] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (media) having exact sequences of instructions, program, code segments, or computer instructions, which is designed and configured to direct, manage, and control the functions of one or more computers or computer systems, and is recorded on a computer- readable medium (media) to prepare a machine, such that the instructions, program, code segments, or computer instructions recorded on the computer-readable medium (media) are used to implement various functions of the system and the method according to the present disclosure.

Claims

1. A speech cloning method based on an autoregressive model, characterized in that, include: Obtain the original text and the corresponding preset reference speech; Using an autoregressive model and self-supervised learning, the preset reference speech is subjected to self-supervised feature extraction to obtain a first feature vector. Style features are extracted from the preset reference speech to obtain a second feature vector. Text features are extracted from the original text to obtain a third feature vector. The simulated speech corresponding to the original text is determined using the autoregressive model based on the first feature vector, the second feature vector, and the third feature vector. Obtain the loss value between the simulated speech and the preset reference speech, and train the autoregressive model based on the loss value; When the loss value meets the preset conditions, the trained autoregressive model is obtained; The current text and the current reference speech are obtained. The autoregressive model is trained and self-supervised feature extraction is performed on the current reference speech in a self-supervised learning manner to obtain the fourth feature vector. Style features are extracted on the current reference speech to obtain the fifth feature vector. Text features are extracted on the current text to obtain the sixth feature vector. The residual decoder in the trained autoregressive model processes the fourth feature vector, the fifth feature vector, and the sixth feature vector to generate the cloned speech corresponding to the current text.

2. The voice cloning method according to claim 1, characterized in that, The process involves using an autoregressive model and self-supervised learning to extract features from the preset reference speech, obtaining a first feature vector; extracting style features from the preset reference speech, obtaining a second feature vector; and extracting text features from the original text, obtaining a third feature vector, including: The first feature vector is obtained by performing self-supervised feature extraction on the preset reference speech through the residual encoder in the autoregressive model in a self-supervised learning manner. The style encoder in the autoregressive model extracts style features from the preset reference speech to obtain a second feature vector; the prior encoder in the autoregressive model extracts text features from the original text to obtain a third feature vector.

3. The voice cloning method according to claim 1, characterized in that, The step of determining the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector includes: The first feature vector, the second feature vector, and the third feature vector are input into the residual decoder in the autoregressive model; The residual decoder in the autoregressive model processes the first feature vector, the second feature vector, and the third feature vector to generate the simulated speech corresponding to the original text.

4. The voice cloning method according to claim 1, characterized in that, The step of obtaining the loss value between the simulated speech and the preset reference speech, and training the autoregressive model based on the loss value, includes: The simulated speech and the preset reference speech are processed by the posterior encoder in the autoregressive model to obtain the audio features of the simulated speech and the audio features of the preset reference speech. The loss function is used to calculate the loss value between the audio features of the simulated speech and the audio features of the preset reference speech, and the autoregressive model is trained based on the loss value.

5. The voice cloning method according to claim 1, characterized in that, The step of obtaining the trained autoregressive model when the loss value meets a preset condition includes: When the loss value is less than a preset value, a stop command is generated; The stop command stops training the autoregressive model, and the trained autoregressive model is saved using a preset file.

6. The voice cloning method according to any one of claims 1 to 5, characterized in that, After obtaining the current text and the current reference speech, and using the trained autoregressive model to perform self-supervised feature extraction on the current reference speech in a self-supervised learning manner to obtain a fourth feature vector, and processing the fourth feature vector to generate the cloned speech corresponding to the current text, the speech cloning method includes: Obtain the display page corresponding to the current text, create a playback window for the display page, and play the cloned voice through the playback window.

7. A speech cloning device based on an autoregressive model, characterized in that, include: The first acquisition module is used to acquire the original text and the preset reference speech corresponding to the original text; The extraction module is used to perform self-supervised feature extraction on the preset reference speech using an autoregressive model in a self-supervised learning manner to obtain a first feature vector, to perform style feature extraction on the preset reference speech to obtain a second feature vector, and to perform text feature extraction on the original text to obtain a third feature vector. The determination module is used to determine the simulated speech corresponding to the original text based on the first feature vector, the second feature vector, and the third feature vector through the autoregressive model; The second acquisition module is used to acquire the loss value between the simulated speech and the preset reference speech, and to train the autoregressive model based on the loss value. The third acquisition module is used to acquire the trained autoregressive model when the loss value meets the preset conditions. The generation module is used to obtain the current text and the current reference speech. Through the trained autoregressive model, the module performs self-supervised feature extraction on the current reference speech in a self-supervised learning manner to obtain the fourth feature vector, performs style feature extraction on the current reference speech to obtain the fifth feature vector, and performs text feature extraction on the current text to obtain the sixth feature vector. The residual decoder in the trained autoregressive model processes the fourth feature vector, the fifth feature vector, and the sixth feature vector to generate the cloned speech corresponding to the current text.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the voice cloning method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the voice cloning method as described in any one of claims 1 to 6.

Images