A voice cloning method, device, equipment and storage medium thereof

Abstract

The embodiment of the application belongs to the field of research and development design and audio processing technology, is applied to different pronunciation object voice conversion processing scenes, and relates to a voice cloning method, device and equipment and a storage medium thereof, voice coding sequences are respectively obtained by inputting source voice and reference voice into a pre-trained self-supervised voice representation model; the voice coding sequence of the source voice is discretized; the discretized voice coding is sequentially used as a query coding, and a target number of most similar codes are screened out from the voice coding sequence of the reference voice; according to the most similar codes and the discretization sequence of all query codes, voice coding sequence cloning is performed to obtain cloned voice coding sequences; the cloned voice coding sequences are input into a preset vocoder to output audio waveforms. The application is applied to voice intelligent customer service and virtual voice generation scenes, and personalized voice selection services are more simply and low-costly provided for customer service or customers.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of research and development design and audio processing technology, and is applied to different pronunciation object voice conversion processing scenes, and particularly relates to a voice cloning method and device, equipment and a storage medium thereof. BACKGROUND

[0002] In recent years, existing neural vocoder technology has made significant progress in the fields of speech synthesis and voice conversion. However, the current technology still has corresponding deficiencies.

[0003] Especially in the field of voice conversion, especially converting source speech into target speech, for example: voice cloning technology, imitating the voice of a specific speaker, such as creating a unique voice for a virtual assistant or voice robot, the current method is mostly an adversarial generation training method, which has high model complexity, high training and inference cost, and cannot provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner. SUMMARY

[0004] The purpose of the embodiments of the present application is to propose a voice cloning method, device, equipment and storage medium to solve the problem that the existing voice conversion field mostly uses an adversarial generation training method, which has high model complexity, high training and inference cost, and cannot provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0005] To solve the above technical problems, the voice cloning method provided by the embodiments of the present application adopts the technical scheme as follows:

[0006] A voice cloning method comprises the following steps:

[0007] inputting a source speech into a pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the source speech, wherein the source speech is the speaking voice of a first speaking object;

[0008] inputting a reference speech into the pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the reference speech, wherein the reference speech is the speaking voice of a second speaking object;

[0009] performing discretization processing on the speech code sequence corresponding to the source speech to obtain a discretized speech code;

[0010] sequentially taking the discretized speech code as a query code to filter out a target number of most similar codes from the speech code sequence corresponding to the reference speech;

[0011] According to the target number of most similar codes corresponding to each query code and the discretization order of all query codes, perform voice code sequence cloning to obtain cloned voice code sequences;

[0012] Input the cloned voice code sequences into a preset vocoder to obtain audio waveforms output by the vocoder.

[0013] Further, before performing the step of inputting the source voice into the pre-trained self-supervised voice representation model to obtain the voice code sequence corresponding to the source voice, the method further comprises:

[0014] Respectively identify the language categories of the source voice and the reference voice;

[0015] By comparison, determine whether the language categories of the source voice and the reference voice are consistent;

[0016] If the language categories of the source voice and the reference voice are inconsistent, input the source voice into a preset voice translator to obtain a translated voice consistent with the language category of the reference voice;

[0017] Update the translated voice as the source voice.

[0018] Further, before performing the step of inputting the source voice into the pre-trained self-supervised voice representation model to obtain the voice code sequence corresponding to the source voice, the method further comprises:

[0019] Based on the language category of the reference voice, obtain a corpus training set of the language category, wherein the corpus training set includes a batch of voices of the language category;

[0020] Input the corpus training set into a preset self-supervised voice representation model for model pre-training to obtain a pre-trained self-supervised voice representation model.

[0021] Further, the step of inputting the corpus training set into a preset self-supervised voice representation model for model pre-training to obtain a pre-trained self-supervised voice representation model specifically comprises:

[0022] Extract the voice code sequence of each training sample in the corpus training set through the preset self-supervised voice representation model;

[0023] Discretize the voice code sequence corresponding to each training sample in the voice training set respectively by using a kmeans method;

[0024] Obtain the discretized voice code contained in each training sample in the voice training set respectively;

[0025] labeling processing is performed on all the discretized speech codes in the speech training set to obtain label values corresponding to all the discretized speech codes respectively;

[0026] The corresponding association relationship between the discretized speech codes and the label values is deployed as pre-training knowledge to the preset self-supervised speech representation model to obtain a pre-trained self-supervised speech representation model.

[0027] Further, the step of inputting the reference speech into the pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the reference speech specifically includes:

[0028] extracting the speech code sequence of the reference speech by using the pre-trained self-supervised speech representation model;

[0029] According to the corresponding association relationship between the discretized speech codes and the label values in the pre-trained self-supervised speech representation model, all the discretized speech codes and the label values contained in the speech code sequence of the reference speech are identified.

[0030] Further, the step of sequentially taking the discretized speech codes as query codes to filter out the target number of most similar codes from the speech code sequence corresponding to the reference speech specifically includes:

[0031] sequentially taking the discretized speech codes contained in the source speech as query codes;

[0032] According to the corresponding association relationship between the discretized speech codes and the label values in the pre-trained self-supervised speech representation model, the label value corresponding to the current query code is identified.

[0033] Using a label value distance comparison algorithm, the target number of discretized speech codes with the smallest label value distance are filtered out from all the discretized speech codes contained in the speech code sequence of the reference speech.

[0034] Further, the step of performing speech code sequence cloning according to the target number of most similar codes corresponding to all the query codes respectively and the discretized order of all the query codes to obtain a cloned speech code sequence specifically includes:

[0035] the target number of most similar codes corresponding to all the query codes respectively are obtained respectively;

[0036] According to the label values of the target number of most similar codes corresponding to the current query code, the current query code is identified by a mode label value;

[0037] If the current query code corresponds to a unique mode label value identification result, the most similar code corresponding to the mode label value is updated as the cloned code of the current query code;

[0038] Otherwise, according to the label value of the target number of most similar codes corresponding to the current query code, the average label value of the current query code is calculated, and the cloned code of the current query code is determined according to the average label value;

[0039] Obtain the cloned code corresponding to each query code, and serialize all cloned codes according to the discretization order of all query codes to obtain the cloned voice code sequence.

[0040] To solve the above technical problems, the embodiment of the application also provides a voice cloning device, which adopts the technical scheme as follows:

[0041] A voice cloning device comprises:

[0042] A source voice serialized code module is configured to input a source voice into a pre-trained self-supervised voice representation model to obtain a voice code sequence corresponding to the source voice, wherein the source voice is the speaking voice of a first speaking object.

[0043] A reference voice serialized code module is configured to input a reference voice into a pre-trained self-supervised voice representation model to obtain a voice code sequence corresponding to the reference voice, wherein the reference voice is the speaking voice of a second speaking object.

[0044] A discretization processing module is configured to perform discretization processing on the voice code sequence corresponding to the source voice to obtain a discretized voice code.

[0045] A most similar code screening module is configured to take the discretized voice code as a query code in turn to screen a target number of most similar codes from the voice code sequence corresponding to the reference voice.

[0046] A voice code sequence cloning module is configured to perform voice code sequence cloning according to the target number of most similar codes corresponding to each query code and the discretization order of all query codes to obtain a cloned voice code sequence.

[0047] A vocoder output module is configured to input the cloned voice code sequence into a pre-set vocoder to obtain an audio waveform output by the vocoder.

[0048] To solve the above technical problems, the embodiment of the application also provides a computer device, which adopts the technical scheme as follows:

[0049] A computer device comprises a memory and a processor, the memory stores computer readable instructions, and the processor implements the steps of the voice cloning method described above when executing the computer readable instructions.

[0050] To solve the above technical problems, the embodiment of the application also provides a computer readable storage medium, which adopts the technical scheme as follows:

[0051] A computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the voice cloning method described above.

[0052] Compared with the prior art, the embodiment of the application has the following beneficial effects:

[0053] The voice cloning method described in the embodiment of the application inputs the source voice into the pre-trained self-supervised speech representation model to obtain the speech code sequence corresponding to the source voice; inputs the reference voice into the pre-trained self-supervised speech representation model to obtain the speech code sequence corresponding to the reference voice; performs discretization processing on the speech code sequence corresponding to the source voice to obtain the discretized speech code; sequentially takes the discretized speech code as a query code to filter out a target number of most similar codes from the speech code sequence corresponding to the reference voice; performs speech code sequence cloning according to the target number of most similar codes corresponding to all query codes respectively and the discretization sequence of all query codes to obtain cloned speech code sequences; and inputs the cloned speech code sequences into a pre-set vocoder to output audio waveforms. The application is applied to the voice intelligent customer service and virtual voice generation scenarios, and more simple and low-cost personalized voice selection services are provided for customer service or customers. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the schemes in the application, the drawings needed in the description of the embodiments of the application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 is an exemplary system architecture diagram to which the application can be applied;

[0056] Figure 2 is a flowchart of one embodiment of the voice cloning method according to the application;

[0057] Figure 3 is a flowchart of one specific embodiment of the preprocessing of the source voice in the voice cloning method described in the application;

[0058] Figure 4 is a flow chart of one specific embodiment of the pre-training of the self-supervised speech representation model in the voice cloning method described in the present application;

[0059] Figure 5 is Figure 4 is a flow chart of one specific embodiment of the step 402 shown in FIG. 4;

[0060] Figure 6 is Figure 2 is a flow chart of one specific embodiment of the step 202 shown in FIG. 2;

[0061] Figure 7 is Figure 2 is a flow chart of one specific embodiment of the step 204 shown in FIG. 2;

[0062] Figure 8 is Figure 2 is a flow chart of one specific embodiment of the step 205 shown in FIG. 2;

[0063] Figure 9 is a structural schematic diagram of one embodiment of the voice cloning apparatus according to the present application;

[0064] Figure 10 is a structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and the drawings are to be regarded as illustrative in nature and are not intended to limit the application; the terminology used in the description and the claims of the present application and the above description of the drawings includes the terms specifically mentioned above, as well as their derivatives.

[0066] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase that in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that the embodiments described herein are merely examples and are not a complete list of alternatives.

[0067] In order to make the technical personnel in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings.

[0068] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102 and a server 103. The terminal device 101 can be a notebook computer 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0069] A user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0070] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer and a desktop computer, etc.

[0071] The server 103 can be a server providing various services, such as a background server supporting the page displayed on the terminal device 101.

[0072] It should be noted that the voice cloning method provided by the embodiments of the present application is generally executed by a server, and correspondingly, the voice cloning device is generally arranged in the server.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in

[0074] With reference to Figure 2 , a flowchart of one embodiment of the voice cloning method according to the present application is shown. The voice cloning method includes the following steps:

[0075] Step 201, input a source voice into a pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the source voice, wherein the source voice is the speaking voice of a first speaking object.

[0076] In this embodiment, the pre-trained self-supervised speech representation model is a model pre-trained according to a speech conversion requirement using training corpus corresponding to the target speech, which can directly obtain the speech coding sequence corresponding to the source speech when actually performing speech conversion processing. In essence, the pre-trained self-supervised speech representation model is a speech encoder pre-trained using a self-supervised training method, for example, a WavLM encoder. Therefore, the self-supervised speech representation model includes a self-supervised speech representation model based on the WavLM encoder.

[0077] By performing sequence speech coding processing on the source speech, the speech coding sequence corresponding to the source speech is obtained, which can facilitate subsequent direct operation processing on the speech coding sequence of the source speech.

[0078] In step 202, the reference speech is input to the pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference speech, wherein the reference speech is the speaking speech of the second speaking object.

[0079] By performing sequence speech coding processing on the reference speech, the speech coding sequence corresponding to the reference speech is obtained, which can facilitate subsequent direct speech cloning processing according to the speech coding sequence of the reference speech. By performing speech coding cloning, the cloned speech coding sequence is obtained, thereby realizing outputting the source speech of the first speaking object as the voice of the second speaking object.

[0080] In step 203, the speech coding sequence corresponding to the source speech is discretized to obtain discretized speech coding.

[0081] In this embodiment, the discretization processing can be implemented using a quantization processor, which can perform quantization processing according to different phonemes involved in the speech coding sequence.

[0082] In step 204, the discretized speech coding is sequentially used as query coding to filter out a target number of most similar codings from the speech coding sequence corresponding to the reference speech.

[0083] In this embodiment, the target number of most similar codings can be filtered out from the speech coding sequence corresponding to the reference speech using a clustering method or a comparison method.

[0084] By sequentially using the discretized speech coding as query coding to filter out a target number of most similar codings from the speech coding sequence corresponding to the reference speech, subsequent speech coding cloning according to the most similar codings can be facilitated.

[0085] In step 205, according to the target number of most similar encodings corresponding to each of all query encodings and the discretization order of all query encodings, perform voice encoding sequence cloning to obtain cloned voice encoding sequences.

[0086] By combining the discretization order of all query encodings, performing voice encoding sequence cloning to obtain cloned voice encoding sequences, the sequence consistency between the cloned voice encoding sequences and the actual voice encoding sequences corresponding to the source voice is ensured, so that the subsequent output voice is still the corresponding sequence of the source voice.

[0087] In step 206, input the cloned voice encoding sequences into a preset vocoder to obtain audio waveforms output by the vocoder.

[0088] In the embodiment, the preset vocoder at least includes frequency domain feature information of the reference voice, and the cloned voice encoding sequences are aligned and embedded with the frequency domain feature information to obtain audio waveforms corresponding to the cloned voice encoding sequences.

[0089] Compared with the existing voice conversion method based on adversarial generation training, the voice cloning method in the embodiment has a simpler model architecture and lower training and inference costs, and can provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0090] In the embodiment, the source voice is input into a pre-trained self-supervised voice representation model to obtain a voice encoding sequence corresponding to the source voice; the reference voice is input into the pre-trained self-supervised voice representation model to obtain a voice encoding sequence corresponding to the reference voice; the voice encoding sequence corresponding to the source voice is discretized to obtain a discretized voice encoding; the discretized voice encoding is sequentially used as a query encoding to filter out a target number of most similar encodings from the voice encoding sequence corresponding to the reference voice; according to the target number of most similar encodings corresponding to each of all query encodings and the discretization order of all query encodings, perform voice encoding sequence cloning to obtain cloned voice encoding sequences; and input the cloned voice encoding sequences into a preset vocoder to output audio waveforms. The voice cloning method can be applied to voice intelligent customer service and virtual voice generation scenarios to provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0091] With reference to Figure 3 In some optional implementations, before step 201, there is also a step of preprocessing the source voice, Figure 3 is a flowchart of a specific embodiment of the voice cloning method described in the present application, including the following steps:

[0092] Step 301, respectively identify the language category of the source voice and the reference voice;

[0093] In this embodiment, the language category includes dialect category, Chinese category, English category, Russian category, French category, etc.

[0094] Step 302, by comparison, determine whether the language category of the source voice and the reference voice is consistent;

[0095] Step 303, if the language category of the source voice and the reference voice is inconsistent, input the source voice into the preset voice translator to obtain a translated voice consistent with the language category of the reference voice;

[0096] Step 304, update the translated voice as the source voice.

[0097] Specifically, before inputting the source voice into the model, the application first determines whether the language category of the source voice and the reference voice is consistent, and processes in combination with the determination result, ensuring that the source voice, the reference voice and the processing voice applicable to the pre-trained self-supervised voice representation model are consistent in language category, avoiding the situation that the voice conversion result is inaccurate due to different language categories.

[0098] Continuing to refer to Figure 4 In some optional implementations, before step 201, there is also a step of pre-training the self-supervised voice representation model, Figure 4 is a flowchart of a specific embodiment of the application for pre-training the self-supervised voice representation model in the voice cloning method, including the following steps:

[0099] Step 401, based on the language category of the reference voice, obtain a corpus training set of the language category, wherein the corpus training set includes a batch of voices of the language category;

[0100] Step 402, input the corpus training set into a preset self-supervised voice representation model for model pre-training to obtain a pre-trained self-supervised voice representation model, wherein the self-supervised voice representation model includes a self-supervised voice representation model based on a WavLM encoder.

[0101] Specifically, when pre-training the self-supervised voice representation model, the embodiment adopts the corpus training set corresponding to the language category of the reference voice for pre-training, ensuring the applicability of the pre-trained self-supervised voice representation model to the reference voice in subsequent actual voice cloning.

[0102] Continuing to refer toFigure 5 , Figure 5 is Figure 4 A flow chart of one specific embodiment of step 402 is shown, including the following steps:

[0103] Step 501, extracting the speech code sequence of each training sample in the training corpus by the preset self-supervised speech representation model;

[0104] Step 502, using the kmeans method to discretize the speech code sequence corresponding to each training sample in the speech training set respectively;

[0105] Specifically, using the kmeans method to discretize the speech code sequence corresponding to each training sample in the speech training set respectively, the number of final label values can be freely set according to the length of the speech code sequence corresponding to each training sample respectively. Assuming that the speech code sequence corresponding to a certain training sample is longer, the number of final label values corresponding to it can be set to be larger. Furthermore, the speech code sequence corresponding to a certain training sample is shorter, the number of final label values corresponding to it can be set to be smaller. Of course, a corresponding control label value quantity can be set in combination with the total data quantity of all training samples in the speech training set, that is, the number of label values set for all training samples in the speech training set does not exceed a certain number, for example: at most 100,000 label values.

[0106] In this embodiment, the kmeans method, i.e. unsupervised method, is used to discretize the speech code sequence corresponding to each training sample in the speech training set, which ensures that the number of label values is more flexible and meets the actual training requirements.

[0107] Step 503, obtaining the discretized speech code contained in each training sample in the speech training set respectively;

[0108] Step 504, performing labelization processing on all discretized speech codes in the speech training set to obtain the label values corresponding to all discretized speech codes respectively;

[0109] Step 505, deploying the corresponding association relationship between the discretized speech code and the label value as pre-training knowledge to the preset self-supervised speech representation model to obtain a pre-trained self-supervised speech representation model.

[0110] Specifically, in this embodiment, the self-supervised speech representation model can be connected to the quantization processor to realize the discretization processing of the speech code sequence corresponding to each training sample in the speech training set. The kmeans algorithm can be used in the quantization processor to perform clustering quantization processing on the speech code sequence.

[0111] By pre-training the self-supervised speech representation model, it is ensured that in subsequent actual speech cloning, the pre-trained self-supervised speech representation model is directly used for speech cloning processing, without the need for re-pre-training in the actual cloning processing stage, thereby saving the processing resource consumption in the actual speech cloning stage.

[0112] With reference to Figure 6 , Figure 6 is Figure 2 a flowchart of one specific embodiment of step 202 shown in FIG. 2, comprising the following steps:

[0113] Step 601, extracting the speech code sequence of the reference speech by the pre-trained self-supervised speech representation model;

[0114] Step 602, identifying all the discretized speech codes and label values contained in the speech code sequence of the reference speech according to the corresponding association relationship between the discretized speech codes and the label values in the pre-trained self-supervised speech representation model.

[0115] By using the pre-trained self-supervised speech representation model to extract the speech code sequence of the reference speech, and identifying all the discretized speech codes and label values contained in the speech code sequence of the reference speech according to the corresponding association relationship between the discretized speech codes and the label values in the pre-trained self-supervised speech representation model, the most similar code can be screened out from all the discretized speech codes contained in the reference speech for subsequent query coding of the source speech.

[0116] With reference to Figure 7 , Figure 7 is Figure 2 a flowchart of one specific embodiment of step 204 shown in FIG. 2, comprising the following steps:

[0117] Step 701, sequentially taking the discretized speech codes contained in the source speech as query codes;

[0118] Step 702, identifying the label value corresponding to the current query code according to the corresponding association relationship between the discretized speech codes and the label values in the pre-trained self-supervised speech representation model;

[0119] Step 703, using a label value distance comparison algorithm to screen out the target number of discretized speech codes with the smallest label value distance from all the discretized speech codes contained in the speech code sequence of the reference speech.

[0120] Specifically, by calculating the label value distance, from all the discretized speech codes contained in the speech code sequence of the reference speech, the target number of discrete speech codes with the smallest label value distance are screened out, ensuring that when the most similar code screening is performed, the most similar code with the closest label value distance and the target number is screened out, to improve the selectability during speech cloning, thereby ensuring the accuracy of the speech cloning result.

[0121] With reference to the foregoing description of the method for selecting the clone code of the query code, Figure 8 , Figure 8 is Figure 2 a flow chart of one specific embodiment of step 205 shown in the figure, comprising the following steps:

[0122] Step 801, respectively acquiring the target number of most similar codes corresponding to all query codes respectively;

[0123] Step 802, according to the label value of the target number of most similar codes corresponding to the current query code, performing mode label value identification on the current query code;

[0124] Step 803, if the mode label value identification result corresponding to the current query code is unique, updating the most similar code corresponding to the mode label value as the clone code of the current query code;

[0125] Step 804, otherwise, according to the label value of the target number of most similar codes corresponding to the current query code, performing average label value calculation on the current query code, and determining the clone code of the current query code according to the average label value;

[0126] Step 805, acquiring the clone codes corresponding to all query codes respectively, and serializing all clone codes according to the discretization order of all query codes, to obtain the cloned speech code sequence.

[0127] Specifically, in the embodiment, when selecting the clone code used by the query code, the mode label value screening method combined with the label value is first adopted, if the clone code cannot be screened out by using the mode label value, the average label value method is used as a backup to screen out the speech code corresponding to the label value closest to the average label value as the clone code of the query code. It is ensured that the selection of the clone code corresponding to the query code is scientific, and by replacing the speech code with the clone code during speech coding, it is ensured that the source speech of the first speaker is output as the voice of the second speaker during speech conversion.

[0128] In the embodiment, the step of determining the cloned code of the current query code according to the average label value specifically comprises: calculating the distance value between the label value of the target number of most similar codes of the current query code and the average label value; and screening the most similar code corresponding to the current query code as the cloned code of the current query code when the distance value is the minimum value.

[0129] Specifically, since the average label value may correspond to a null speech code, the distance value between the label value of the target number of most similar codes of the current query code and the average label value is calculated, and the most similar code corresponding to the current query code is screened as the cloned code of the current query code when the distance value is the minimum value, so that the cloned code can be screened from the target number of most similar codes, and the process is more scientific.

[0130] The application obtains a speech code sequence corresponding to the source speech by inputting the source speech into a pre-trained self-supervised speech representation model, obtains a speech code sequence corresponding to the reference speech by inputting the reference speech into the pre-trained self-supervised speech representation model, performs discretization processing on the speech code sequence corresponding to the source speech to obtain a discretized speech code, and sequentially takes the discretized speech code as a query code to screen a target number of most similar codes from the speech code sequence corresponding to the reference speech. The application performs speech code sequence cloning according to the target number of most similar codes corresponding to all query codes respectively and the discretization sequence of all query codes, obtains a cloned speech code sequence, and inputs the cloned speech code sequence into a preset vocoder to output an audio waveform. The application is applied to a voice intelligent customer service and a virtual voice generation scene, and can more simply and low-costly provide personalized voice selection services for a customer service or a customer.

[0131] The embodiments of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0132] Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and the like. Artificial intelligence software technologies mainly include computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning and the like.

[0133] In the embodiment of the application, the source speech is input into a pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the source speech; the reference speech is input into the pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the reference speech; the speech code sequence corresponding to the source speech is discretized to obtain a discretized speech code; the discretized speech code is sequentially taken as a query code to filter out a target number of most similar codes from the speech code sequence corresponding to the reference speech; speech code sequence cloning is performed according to the target number of most similar codes corresponding to all query codes respectively and the discretization order of all query codes to obtain cloned speech code sequences; and the cloned speech code sequences are input into a pre-set vocoder to output audio waveforms. The application is applied to a voice intelligent customer service and a virtual voice generation scene to provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0134] Further reference Figure 9 , as an implementation of the method shown in the above Figure 2 , the application provides an embodiment of a voice cloning device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0135] As shown in Figure 9 , the voice cloning device 900 described in the embodiment includes a source speech serialization coding module 901, a reference speech serialization coding module 902, a discretization processing module 903, a most similar code filtering module 904, a speech code sequence cloning module 905, and a vocoder output module 906.

[0136] Among them:

[0137] The source speech serialization coding module 901 is configured to input source speech into a pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the source speech, wherein the source speech is the speaking voice of a first speaking object.

[0138] The reference speech serialization coding module 902 is configured to input reference speech into a pre-trained self-supervised speech representation model to obtain a speech code sequence corresponding to the reference speech, wherein the reference speech is the speaking voice of a second speaking object.

[0139] The discretization processing module 903 is configured to discretize the speech code sequence corresponding to the source speech to obtain a discretized speech code.

[0140] The most similar code filtering module 904 is configured to sequentially take the discretized speech code as a query code to filter out a target number of most similar codes from the speech code sequence corresponding to the reference speech.

[0141] The voice coding sequence cloning module 905 is configured to clone the voice coding sequence according to the target number of most similar codes corresponding to all the query codes respectively and the discretization sequence of all the query codes, to obtain a cloned voice coding sequence.

[0142] The vocoder output module 906 is configured to input the cloned voice coding sequence into a preset vocoder, to obtain an audio waveform output by the vocoder.

[0143] The voice coding sequence cloning module 905 is configured to clone the voice coding sequence according to the target number of most similar codes corresponding to all the query codes respectively and the discretization sequence of all the query codes, to obtain a cloned voice coding sequence.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0145] It should be understood that, although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0146] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 10 , Figure 10 The basic structure block diagram of the computer device of the embodiment is shown in the figure.

[0147] The computer device 10 comprises a memory 10a, a processor 10b, and a network interface 10c which are connected to each other through a system bus. It should be pointed out that, Figure 10 The computer device 10 with the components of the memory 10a, the processor 10b, and the network interface 10c is only shown in the figure, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented. Among them, the computer device herein is a device which can automatically perform numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0148] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can perform human-computer interaction with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and the like.

[0149] The memory 10a includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or a memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 10. Of course, the memory 10a can also include both an internal storage unit and an external storage device of the computer device 10. In this embodiment, the memory 10a is generally used to store an operating system and various application software installed on the computer device 10, such as computer readable instructions of a voice cloning method, etc. In addition, the memory 10a can also be used to temporarily store various data that have been output or will be output.

[0150] The processor 10b can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run computer readable instructions or process data stored in the memory 10a, such as computer readable instructions of the voice cloning method.

[0151] The network interface 10c can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0152] The computer device provided in the embodiment belongs to the technical field of research and development design and audio processing, and is applied to different voice conversion processing scenes of different pronunciation objects. The voice cloning method provided in the embodiment comprises the following steps: inputting source voice into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the source voice; inputting reference voice into the pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference voice; performing discretization processing on the speech coding sequence corresponding to the source voice to obtain discretized speech coding; taking the discretized speech coding as query coding in sequence to filter out a target number of most similar codes from the speech coding sequence corresponding to the reference voice; performing speech coding sequence cloning according to the target number of most similar codes corresponding to all query codes respectively and the discretization sequence of all query codes to obtain cloned speech coding sequences; and inputting the cloned speech coding sequences into a preset vocoder to output audio waveforms. The voice cloning method is applied to voice intelligent customer service and virtual voice generation scenes, and can provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0153] The voice cloning method provided in the embodiment is applied to different voice conversion processing scenes of different pronunciation objects. The voice cloning method comprises the following steps: inputting source voice into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the source voice; inputting reference voice into the pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference voice; performing discretization processing on the speech coding sequence corresponding to the source voice to obtain discretized speech coding; taking the discretized speech coding as query coding in sequence to filter out a target number of most similar codes from the speech coding sequence corresponding to the reference voice; performing speech coding sequence cloning according to the target number of most similar codes corresponding to all query codes respectively and the discretization sequence of all query codes to obtain cloned speech coding sequences; and inputting the cloned speech coding sequences into a preset vocoder to output audio waveforms. The voice cloning method is applied to voice intelligent customer service and virtual voice generation scenes, and can provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0154] The voice cloning method provided in the embodiment is applied to different voice conversion processing scenes of different pronunciation objects. The voice cloning method comprises the following steps: inputting source voice into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the source voice; inputting reference voice into the pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference voice; performing discretization processing on the speech coding sequence corresponding to the source voice to obtain discretized speech coding; taking the discretized speech coding as query coding in sequence to filter out a target number of most similar codes from the speech coding sequence corresponding to the reference voice; performing speech coding sequence cloning according to the target number of most similar codes corresponding to all query codes respectively and the discretization sequence of all query codes to obtain cloned speech coding sequences; and inputting the cloned speech coding sequences into a preset vocoder to output audio waveforms. The voice cloning method is applied to voice intelligent customer service and virtual voice generation scenes, and can provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0155] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0156] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A voice cloning method, characterized in that, Includes the following steps: The source speech is input into a pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, wherein the source speech is the speech of the first speaker; The reference speech is input into a pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker; The speech coding sequence corresponding to the source speech is discretized to obtain the discretized speech coding; The discrete speech codes are used sequentially as query codes to select the target number of most similar codes from the speech code sequence corresponding to the reference speech; Based on the most similar code corresponding to the target number of each query code and the discretization order of all query codes, the speech coding sequence is cloned to obtain the cloned speech coding sequence. The cloned speech encoding sequence is input into a preset vocoder to obtain the audio waveform output by the vocoder.

2. The voice cloning method according to claim 1, characterized in that, Before performing the step of inputting the source speech into a pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, the method further includes: Identify the language categories of the source speech and the reference speech respectively; By comparison, it is determined whether the language categories of the source speech and the reference speech are consistent; If the source speech and the reference speech are not in the same language category, the source speech is input into a preset speech translator to obtain a translated speech that is consistent with the language category of the reference speech. Update the translated speech to the source speech.

3. The voice cloning method according to claim 1, characterized in that, Before performing the step of inputting the source speech into a pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, the method further includes: Based on the language category of the reference speech, a training set of speech data for the language category is obtained, wherein the training set of speech data includes a batch of speech data for the language category. The training corpus is input into a pre-defined self-supervised speech representation model for pre-training to obtain a pre-trained self-supervised speech representation model.

4. The voice cloning method according to claim 3, characterized in that, The step of inputting the corpus training set into a preset self-supervised speech representation model for model pre-training to obtain a pre-trained self-supervised speech representation model specifically includes: The speech encoding sequence of each training sample in the corpus training set is extracted using the preset self-supervised speech representation model. The kmeans method is used to discretize the speech coding sequence corresponding to each training sample in the speech training set. Obtain the discrete speech code contained in each training sample in the speech training set; All discrete speech codes in the speech training set are labeled to obtain the label values ​​corresponding to each discrete speech code. The correspondence between the discretized speech code and the label value is used as pre-trained knowledge and deployed to the preset self-supervised speech representation model to obtain the pre-trained self-supervised speech representation model.

5. The voice cloning method according to claim 4, characterized in that, The step of inputting the reference speech into a pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech specifically includes: The speech coding sequence of the reference speech is extracted using the pre-trained self-supervised speech representation model. Based on the correspondence between the discrete speech codes and label values ​​in the pre-trained self-supervised speech representation model, all discrete speech codes and label values ​​contained in the speech coding sequence of the reference speech are identified.

6. The voice cloning method according to claim 4, characterized in that, The step of sequentially using the discretized speech codes as query codes to select the target number of most similar codes from the speech code sequence corresponding to the reference speech specifically includes: The discrete speech codes contained in the source speech are used as query codes in sequence; Based on the correspondence between the discretized speech code and the label value in the pre-trained self-supervised speech representation model, the label value corresponding to the current query code is identified. A label value distance comparison algorithm is used to select the discrete speech code with the smallest label value distance from all discrete speech codes contained in the speech coding sequence of the reference speech.

7. The voice cloning method according to claim 1, characterized in that, The step of cloning the speech coding sequence based on the most similar code corresponding to the target number of each query code and the discretization order of all query codes, and obtaining the cloned speech coding sequence, specifically includes: Obtain the number of most similar codes corresponding to the target for each query code; Based on the label value of the most similar code to the target number corresponding to the current query code, the mode label value is identified for the current query code; If the identification result of the mode label value corresponding to the current query code is unique, then the most similar code corresponding to the mode label value is updated to the clone code of the current query code; Otherwise, based on the label value of the most similar code to the target number corresponding to the current query code, the average label value of the current query code is calculated, and the clone code of the current query code is determined based on the average label value; Obtain the clone codes corresponding to all query codes, and serialize and organize all clone codes according to the discretization order of all query codes to obtain the cloned speech code sequence.

8. A voice cloning device, characterized in that, include: The source speech serialization encoding module is used to input the source speech into a pre-trained self-supervised speech representation model to obtain the speech encoding sequence corresponding to the source speech, wherein the source speech is the speech of the first speaker; The reference speech serialization encoding module is used to input the reference speech into a pre-trained self-supervised speech representation model to obtain the speech encoding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker; The discretization processing module is used to discretize the speech coding sequence corresponding to the source speech to obtain discretized speech coding; The most similar code filtering module is used to sequentially use the discrete speech codes as query codes to filter out the target number of most similar codes from the speech code sequence corresponding to the reference speech; The speech coding sequence cloning module is used to clone the speech coding sequence based on the most similar code of the target number corresponding to each of the query codes and the discretization order of all query codes, and to obtain the cloned speech coding sequence. The vocoder output module is used to input the cloned speech encoding sequence into a preset vocoder and obtain the audio waveform output by the vocoder.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the voice cloning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech cloning method as described in any one of claims 1 to 7.

Images