Voice cloning method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of research and development design and audio processing, is applied to voice conversion processing scenes of different pronunciation objects, and relates to a voice cloning method, device and equipment and a storage medium thereof.The voice cloning method comprises the steps that source voice and reference voice are input into a pre-trained self-supervised voice representation model, and voice coding sequences are obtained respectively; discretizing the voice coding sequence of the source voice; taking the discretized voice codes as query codes in sequence, and screening out a target number of most similar codes from the voice code sequence of the reference voice; according to the most similar codes and the discretization sequence of all query codes, voice coding sequence cloning is carried out, and a cloned voice coding sequence is obtained; and inputting the cloned voice coding sequence into a preset vocoder, and outputting an audio waveform. The method is applied to voice intelligent customer service and virtual voice generation scenes, and personalized voice selection services are provided for customer service or customers more simply at low cost.

Description

Technical Field

[0001] The present application relates to the field of R&D design and audio processing technology, and is applied to speech conversion processing scenarios of different pronunciation objects, and in particular to a speech cloning method, device, equipment and storage medium thereof. Background Art

[0002] In recent years, existing neural vocoder technology has made significant progress in fields such as speech synthesis and speech conversion. However, the current technology still has corresponding shortcomings.

[0003] Especially in the field of voice conversion, especially converting source voice into target voice, such as voice cloning technology, imitating the voice of a specific speaker, such as creating a unique voice for a virtual assistant or voice robot. At present, adversarial generation training is mostly used, which has the problems of high model complexity, high training and inference costs, and the inability to provide personalized voice selection services to customer service or customers in a simpler and lower-cost manner. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a voice cloning method, apparatus, device and storage medium thereof to solve the problem that in the existing field of voice conversion, adversarial generation training is mostly used, the model complexity is high, the training and reasoning costs are high, and it is impossible to provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0005] In order to solve the above technical problems, the present application embodiment provides a voice cloning method, which adopts the following technical solution:

[0006] A voice cloning method comprises the following steps:

[0007] Inputting a source speech into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the source speech, wherein the source speech is a speech of a first speaker;

[0008] Inputting a reference speech into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker;

[0009] Discretizing the speech coding sequence corresponding to the source speech to obtain a discretized speech coding;

[0010] The discretized speech codes are used as query codes in sequence, and the target number of most similar codes are selected from the speech code sequence corresponding to the reference speech;

[0011] According to the most similar codes of the target number corresponding to all query codes respectively and the discretization order of all query codes, the voice code sequence is cloned to obtain the cloned voice code sequence;

[0012] The cloned speech coding sequence is input into a preset vocoder to obtain an audio waveform output by the vocoder.

[0013] Furthermore, before executing the step of inputting the source speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, the method further includes:

[0014] Respectively identifying the language categories of the source speech and the reference speech;

[0015] By comparison, determining whether the language category of the source speech and the reference speech is consistent;

[0016] If the language categories of the source speech and the reference speech are inconsistent, the source speech is input into a preset speech translator to obtain a translated speech that is consistent with the language category of the reference speech;

[0017] The translated speech is updated to the source speech.

[0018] Furthermore, before executing the step of inputting the source speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, the method further includes:

[0019] Based on the language category of the reference speech, obtaining a corpus training set of the language category, wherein the corpus training set includes batches of speech of the language category;

[0020] The corpus training set is input into a preset self-supervised speech representation model to perform model pre-training to obtain a pre-trained self-supervised speech representation model.

[0021] Furthermore, the step of inputting the corpus training set into a preset self-supervised speech representation model, performing model pre-training, and obtaining a pre-trained self-supervised speech representation model specifically includes:

[0022] Extracting the speech coding sequence of each training sample in the corpus training set by using the preset self-supervised speech representation model;

[0023] Using the kmeans method, discretizing the speech coding sequence corresponding to each training sample in the speech training set;

[0024] Obtaining the discretized speech code contained in each training sample in the speech training set;

[0025] Performing labeling processing on all the discretized speech codes in the speech training set to obtain label values ​​corresponding to all the discretized speech codes;

[0026] The corresponding association relationship between the discretized speech coding and the label value is deployed as pre-trained knowledge to the preset self-supervised speech representation model to obtain the pre-trained self-supervised speech representation model.

[0027] Furthermore, the step of inputting the reference speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech specifically includes:

[0028] Extracting the speech coding sequence of the reference speech through the pre-trained self-supervised speech representation model;

[0029] According to the corresponding association relationship between the discretized speech codes and the label values ​​in the pre-trained self-supervised speech representation model, all the discretized speech codes and label values ​​contained in the speech code sequence of the reference speech are identified.

[0030] Furthermore, the step of using the discretized speech codes as query codes in sequence and selecting the target number of most similar codes from the speech code sequence corresponding to the reference speech specifically includes:

[0031] sequentially taking the discretized speech codes contained in the source speech as query codes;

[0032] Identifying the label value corresponding to the current query code according to the corresponding association relationship between the discretized speech code and the label value in the pre-trained self-supervised speech representation model;

[0033] A label value distance comparison algorithm is adopted to screen out the discretized speech codes with the smallest label value distance of the target number from all the discretized speech codes included in the speech code sequence of the reference speech.

[0034] Furthermore, the step of cloning the speech coding sequence according to the most similar codes of the target number corresponding to all query codes and the discretization order of all query codes to obtain the cloned speech coding sequence specifically includes:

[0035] Respectively obtain the most similar codes of the target quantity corresponding to all query codes;

[0036] According to the label value of the most similar code of the target number corresponding to the current query code, the majority label value of the current query code is identified;

[0037] If the majority tag value identification result corresponding to the current query code is unique, then the most similar code corresponding to the majority tag value is updated to be the clone code of the current query code;

[0038] Otherwise, according to the label values ​​of the most similar codes of the target number corresponding to the current query code, an average label value is calculated for the current query code, and a clone code of the current query code is determined according to the average label value;

[0039] The clone codes corresponding to all query codes are obtained, and all clone codes are serialized and sorted according to the discretization order of all query codes to obtain the cloned speech code sequence.

[0040] In order to solve the above technical problems, the embodiment of the present application also provides a voice cloning device, which adopts the following technical solution:

[0041] A voice cloning device, comprising:

[0042] A source speech serialization encoding module, used for inputting a source speech into a pre-trained self-supervised speech representation model to obtain a speech encoding sequence corresponding to the source speech, wherein the source speech is the speech of the first speaker;

[0043] A reference speech serialization encoding module, used for inputting a reference speech into a pre-trained self-supervised speech representation model to obtain a speech encoding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker;

[0044] A discretization processing module, used for discretizing the speech coding sequence corresponding to the source speech to obtain a discretized speech coding;

[0045] A most similar code screening module, used to use the discretized speech codes as query codes in sequence, and screen out a target number of most similar codes from the speech code sequence corresponding to the reference speech;

[0046] A speech coding sequence cloning module, used to clone the speech coding sequence according to the most similar codes of the target number corresponding to all query codes respectively and the discretization order of all query codes, and obtain the cloned speech coding sequence;

[0047] The vocoder output module is used to input the cloned speech coding sequence into a preset vocoder to obtain the audio waveform output by the vocoder.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0049] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned voice cloning method when executing the computer-readable instructions.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0051] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the voice cloning method as described above.

[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0053] The voice cloning method described in the embodiment of the present application obtains the voice coding sequence corresponding to the source voice by inputting the source voice into the pre-trained self-supervised voice representation model; inputs the reference voice into the pre-trained self-supervised voice representation model to obtain the voice coding sequence corresponding to the reference voice; discretizes the voice coding sequence corresponding to the source voice to obtain discretized voice coding; uses the discretized voice coding as query coding in turn to screen out the target number of most similar codings from the voice coding sequence corresponding to the reference voice; clones the voice coding sequence according to the most similar coding of the target number corresponding to all query codings, and the discretization order of all query codings, to obtain the cloned voice coding sequence; inputs the cloned voice coding sequence into a preset vocoder to output an audio waveform. Applied to voice intelligent customer service and virtual voice generation scenarios, it is simpler and more cost-effective to provide personalized voice selection services for customer service or customers. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flow chart of an embodiment of a voice cloning method according to the present application;

[0057] Figure 3 It is a flowchart of a specific embodiment of preprocessing the source voice in the voice cloning method described in the present application;

[0058] Figure 4 It is a flowchart of a specific embodiment of pre-training a self-supervised speech representation model in the speech cloning method described in the present application;

[0059] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 is shown;

[0060] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0061] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;

[0062] Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0063] Fig. 9 is a structural schematic diagram of an embodiment of a voice cloning device according to the present application;

[0064] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0069] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0070] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0071] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0072] It should be noted that the voice cloning method provided in the embodiment of the present application is generally executed by a server, and accordingly, the voice cloning device is generally disposed in the server.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0074] Continue to refer Figure 2 , shows a flow chart of an embodiment of a voice cloning method according to the present application. The voice cloning method comprises the following steps:

[0075] Step 201: input source speech into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the source speech, wherein the source speech is the speech of a first speaker.

[0076] In this embodiment, the pre-trained self-supervised speech representation model is a model pre-trained using training corpus corresponding to the target speech according to speech conversion requirements, and it can directly obtain the speech coding sequence corresponding to the source speech when the speech conversion process is actually performed. In essence, the pre-trained self-supervised speech representation model is a speech encoder pre-trained using a self-supervised training method, such as a WavLM encoder, and therefore, the self-supervised speech representation model includes a self-supervised speech representation model based on the WavLM encoder.

[0077] By performing serialized speech coding processing on the source speech, a speech coding sequence corresponding to the source speech is obtained, which can facilitate subsequent direct operation and processing on the speech coding sequence of the source speech.

[0078] Step 202: input a reference speech into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker.

[0079] By performing serialized speech coding processing on the reference speech to obtain a speech coding sequence corresponding to the reference speech, it is convenient to perform speech cloning processing directly according to the speech coding sequence of the reference speech. By performing speech coding cloning, a cloned speech coding sequence is obtained, thereby outputting the source speech of the first speaker as the voice of the second speaker.

[0080] Step 203: discretize the speech code sequence corresponding to the source speech to obtain a discretized speech code.

[0081] In this embodiment, the discretization process can be implemented by using a quantization processor. The quantization processor can perform quantization process according to the differences of phonemes involved in the speech coding sequence.

[0082] Step 204: using the discretized speech codes as query codes in sequence, and selecting the target number of most similar codes from the speech code sequence corresponding to the reference speech.

[0083] In this embodiment, the method of selecting the target number of most similar codes from the speech coding sequence corresponding to the reference speech may adopt a clustering method to select the target number of most similar codes, or may adopt a comparison method to select the target number of most similar codes.

[0084] By using the discretized speech codes as query codes in turn, a target number of most similar codes are screened out from the speech code sequence corresponding to the reference speech, so as to facilitate subsequent speech code cloning according to the most similar codes.

[0085] Step 205 , cloning the speech code sequence according to the most similar codes of the target number corresponding to all query codes respectively and the discretization order of all query codes, and obtaining the cloned speech code sequence.

[0086] By combining the discretization order of all query codes, the voice coding sequence is cloned to obtain the cloned voice coding sequence, which ensures the sequence consistency between the cloned voice coding sequence and the actual voice coding sequence corresponding to the source voice, thereby ensuring that the subsequent output voice is still the corresponding sequence of the source voice.

[0087] Step 206: input the cloned speech coding sequence into a preset vocoder to obtain an audio waveform output by the vocoder.

[0088] In this embodiment, the preset vocoder includes at least the frequency domain feature information of the reference speech, and the audio waveform corresponding to the cloned speech coding sequence is obtained by aligning and embedding the cloned speech coding sequence with the frequency domain feature information.

[0089] The voice cloning method described in this embodiment is applied to voice intelligent customer service and virtual voice generation scenarios. Compared with the existing voice conversion method based on adversarial generation training, the model architecture is relatively simple, and the training and inference costs are low, ensuring a simpler and lower-cost personalized voice selection service for customer service or customers.

[0090] In this embodiment, the source speech is input into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech; the reference speech is input into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech; the speech coding sequence corresponding to the source speech is discretized to obtain the discretized speech coding; the discretized speech coding is used as the query coding in turn to screen out the target number of most similar codings from the speech coding sequence corresponding to the reference speech; the speech coding sequence is cloned according to the most similar codings of the target number corresponding to all query codings and the discretization order of all query codings to obtain the cloned speech coding sequence; the cloned speech coding sequence is input into the preset vocoder to output the audio waveform. It is applied to the scenarios of voice intelligent customer service and virtual speech generation to provide personalized voice selection services for customer service or customers in a simpler and lower-cost manner.

[0091] Continue to refer Figure 3 In some optional implementations, a step of preprocessing the source speech is also included before step 201. Figure 3 This is a flowchart of a specific embodiment of preprocessing the source voice in the voice cloning method described in the present application, comprising the following steps:

[0092] Step 301, identifying the language categories of the source speech and the reference speech respectively;

[0093] In this embodiment, the language category includes dialect category, Chinese category, English category, Russian category, French category, etc.

[0094] Step 302, determining whether the language category of the source speech and the reference speech is consistent through comparison;

[0095] Step 303: if the language categories of the source speech and the reference speech are inconsistent, the source speech is input into a preset speech translator to obtain a translated speech that is consistent with the language category of the reference speech;

[0096] Step 304: Update the translated speech to the source speech.

[0097] Specifically, before inputting the source speech into the model, the present application first determines whether the language categories of the source speech and the reference speech are consistent, and performs processing based on the judgment result, thereby ensuring that the source speech, the reference speech and the processed speech applicable to the pre-trained self-supervised speech representation model are consistent in language category, thereby avoiding the occurrence of inaccurate speech conversion results due to different language categories.

[0098] Continue to refer Figure 4 In some optional implementations, a step of pre-training the self-supervised speech representation model is also included before step 201. Figure 4 It is a flowchart of a specific embodiment of pre-training a self-supervised speech representation model in the speech cloning method described in the present application, comprising the following steps:

[0099] Step 401, based on the language category of the reference speech, obtaining a corpus training set of the language category, wherein the corpus training set includes batch speech of the language category;

[0100] Step 402, input the corpus training set into a preset self-supervised speech representation model, perform model pre-training, and obtain a pre-trained self-supervised speech representation model, wherein the self-supervised speech representation model includes a self-supervised speech representation model based on a WavLM encoder.

[0101] Specifically, this embodiment uses a corpus training set corresponding to the language category of the reference speech for pre-training the self-supervised speech representation model, thereby ensuring the applicability of the pre-trained self-supervised speech representation model to the processing of the reference speech during subsequent actual speech cloning.

[0102] Continue to refer Figure 5 , Figure 5 yes Figure 4 The flowchart of a specific embodiment of step 402 shown includes the following steps:

[0103] Step 501, extracting the speech coding sequence of each training sample in the corpus training set through the preset self-supervised speech representation model;

[0104] Step 502, using the kmeans method to discretize the speech coding sequence corresponding to each training sample in the speech training set;

[0105] Specifically, the kmeans method is used to discretize the speech coding sequence corresponding to each training sample in the speech training set, and the number of final label values ​​can be freely set according to the length of the speech coding sequence corresponding to each training sample. Suppose, if the speech coding sequence corresponding to a certain training sample is longer, the number of its corresponding final label values ​​can be set to be larger. Furthermore, if the speech coding sequence corresponding to a certain training sample is shorter, the number of its corresponding final label values ​​can be set to be smaller. Of course, a corresponding controlling number of label values ​​can be set in combination with the total data volume of all training samples in the speech training set, that is, the number of label values ​​set for all training samples in the speech training set does not exceed a certain number, for example: a maximum of 100,000 label values ​​can be set.

[0106] In this embodiment, the kmeans method is used, that is, the speech coding sequence corresponding to each training sample in the speech training set is discretized in an unsupervised manner, which ensures that the number of label values ​​is set more flexibly and in line with actual training requirements.

[0107] Step 503, obtaining the discretized speech code contained in each training sample in the speech training set;

[0108] Step 504, labeling all the discretized speech codes in the speech training set to obtain label values ​​corresponding to all the discretized speech codes;

[0109] Step 505: deploy the corresponding association relationship between the discretized speech coding and the label value as pre-trained knowledge to the preset self-supervised speech representation model to obtain a pre-trained self-supervised speech representation model.

[0110] Specifically, in this embodiment, when the self-supervised speech representation model is pre-trained, the quantization processor can be connected to realize discretization processing of the speech coding sequence corresponding to each training sample in the speech training set, and the kmeans algorithm can be used in the quantization processor to perform clustering quantization processing on the speech coding sequence.

[0111] By pre-training the self-supervised speech representation model, it is ensured that in the subsequent actual speech cloning, the pre-trained self-supervised speech representation model is directly used for speech cloning processing, without the need for pre-training again in the actual cloning processing stage, thus saving processing resource consumption in the actual speech cloning stage.

[0112] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0113] Step 601, extracting the speech coding sequence of the reference speech through the pre-trained self-supervised speech representation model;

[0114] Step 602: According to the corresponding association relationship between the discretized speech codes and the label values ​​in the pre-trained self-supervised speech representation model, all the discretized speech codes and label values ​​included in the speech code sequence of the reference speech are identified.

[0115] By adopting the pre-trained self-supervised speech representation model, the speech coding sequence of the reference speech is extracted, and according to the corresponding association relationship between the discretized speech coding and the label value in the pre-trained self-supervised speech representation model, all the discretized speech coding and label values ​​contained in the speech coding sequence of the reference speech are identified, so that the query coding of the subsequent source speech can filter out the most similar coding from all the discretized speech coding contained in the reference speech.

[0116] Continue to refer Figure 7 , Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes the following steps:

[0117] Step 701, sequentially using the discretized speech codes contained in the source speech as query codes;

[0118] Step 702, identifying the label value corresponding to the current query code according to the corresponding association relationship between the discretized speech code and the label value in the pre-trained self-supervised speech representation model;

[0119] Step 703: Using a label value distance comparison algorithm, select the discretized speech codes with the smallest label value distance of the target number from all the discretized speech codes included in the speech code sequence of the reference speech.

[0120] Specifically, by calculating the label value distance, the discretized speech codes with the smallest label value distance of the target number are screened out from all the discretized speech codes contained in the speech code sequence of the reference speech, ensuring that when the most similar codes are screened out, the most similar codes with the shortest label value distance and the target number are screened out, so as to improve the selectivity during speech cloning, thereby ensuring the accuracy of the speech cloning result.

[0121] Continue to refer Figure 8 , Figure 8 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes the following steps:

[0122] Step 801, respectively obtaining the most similar codes of the target quantity corresponding to all query codes;

[0123] Step 802, identifying the majority label value of the current query code according to the label values ​​of the most similar codes of the target number corresponding to the current query code;

[0124] Step 803: if the majority tag value identification result corresponding to the current query code is unique, then the most similar code corresponding to the majority tag value is updated as a clone code of the current query code;

[0125] Step 804: otherwise, calculate the average label value of the current query code according to the label values ​​of the target number of most similar codes corresponding to the current query code, and determine the cloned codes of the current query code according to the average label value;

[0126] Step 805: Obtain the clone codes corresponding to all query codes respectively, and serialize and sort all clone codes according to the discretization order of all query codes to obtain the cloned speech code sequence.

[0127] Specifically, in this embodiment, when selecting the clone code finally used for the query code, the majority tag value screening method is first adopted in combination with the tag value. If the majority tag value cannot screen out the clone code, the average tag value method is used as a backup to screen out the voice code corresponding to the tag value closest to the average tag value as the clone code of the query code. The scientific nature of the selection of the clone code corresponding to the query code is ensured. By replacing the clone code with the clone code during the voice coding, it is ensured that during the voice conversion, the source voice of the first speaker is output as the voice of the second speaker.

[0128] In this embodiment, the step of determining the clone code of the current query code based on the average label value specifically includes: calculating the label values ​​of the target number of most similar codes of the current query code, and the distance values ​​between them and the average label value; when the distance value is the minimum, the most similar code corresponding to the current query code is used as the clone code of the current query code.

[0129] Specifically, since the average label value may correspond to a null value in the speech code, the label values ​​of the target number of most similar codes of the current query code are calculated, and their distance values ​​from the average label value are respectively calculated; when the distance value is the minimum, the most similar code corresponding to the current query code is used as the clone code of the current query code, which ensures that the clone code can be screened out from the most similar codes of the target number, which is more scientific.

[0130] This application obtains the speech coding sequence corresponding to the source speech by inputting the source speech into the pre-trained self-supervised speech representation model; inputting the reference speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech; discretizing the speech coding sequence corresponding to the source speech to obtain the discretized speech coding; using the discretized speech coding as the query coding in turn, and screening out the target number of most similar codings from the speech coding sequence corresponding to the reference speech; cloning the speech coding sequence according to the most similar coding of the target number corresponding to all query codings, and the discretization order of all query codings, to obtain the cloned speech coding sequence; inputting the cloned speech coding sequence into the preset vocoder to output the audio waveform. Applied to the scenarios of voice intelligent customer service and virtual speech generation, it is simpler and more cost-effective to provide personalized voice selection services for customer service or customers.

[0131] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0132] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0133] In the embodiment of the present application, the source speech is input into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech; the reference speech is input into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech; the speech coding sequence corresponding to the source speech is discretized to obtain the discretized speech coding; the discretized speech coding is used as the query coding in turn to screen out the target number of most similar codings from the speech coding sequence corresponding to the reference speech; the speech coding sequence is cloned according to the most similar codings of the target number corresponding to all query codings, and the discretization order of all query codings, to obtain the cloned speech coding sequence; the cloned speech coding sequence is input into the preset vocoder to output the audio waveform. Applied to the scenarios of voice intelligent customer service and virtual speech generation, it is simpler and more cost-effective to provide personalized voice selection services for customer service or customers.

[0134] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a voice cloning device, and the embodiment of the device is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0135] like Fig. 9 As shown, the speech cloning device 900 described in this embodiment includes: a source speech serialization encoding module 901, a reference speech serialization encoding module 902, a discretization processing module 903, a most similar encoding screening module 904, a speech coding sequence cloning module 905 and a vocoder output module 906.

[0136] in:

[0137] A source speech serialization encoding module 901 is used to input a source speech into a pre-trained self-supervised speech representation model to obtain a speech encoding sequence corresponding to the source speech, wherein the source speech is the speech of the first speaker;

[0138] A reference speech serialization encoding module 902 is used to input a reference speech into a pre-trained self-supervised speech representation model to obtain a speech encoding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker;

[0139] A discretization processing module 903 is used to discretize the speech code sequence corresponding to the source speech to obtain a discretized speech code;

[0140] The most similar code screening module 904 is used to use the discretized speech codes as query codes in sequence, and screen out a target number of most similar codes from the speech code sequence corresponding to the reference speech;

[0141] The speech coding sequence cloning module 905 is used to clone the speech coding sequence according to the most similar codes of the target number corresponding to all query codes and the discretization order of all query codes, and obtain the cloned speech coding sequence;

[0142] The vocoder output module 906 is used to input the cloned speech coding sequence into a preset vocoder to obtain an audio waveform output by the vocoder.

[0143] This application obtains the speech coding sequence corresponding to the source speech by inputting the source speech into the pre-trained self-supervised speech representation model; inputting the reference speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech; discretizing the speech coding sequence corresponding to the source speech to obtain the discretized speech coding; using the discretized speech coding as the query coding in turn, and screening out the target number of most similar codings from the speech coding sequence corresponding to the reference speech; cloning the speech coding sequence according to the most similar coding of the target number corresponding to all query codings, and the discretization order of all query codings, to obtain the cloned speech coding sequence; inputting the cloned speech coding sequence into the preset vocoder to output the audio waveform. Applied to the scenarios of voice intelligent customer service and virtual speech generation, it is simpler and more cost-effective to provide personalized voice selection services for customer service or customers.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0145] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0146] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10 Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0148] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0149] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store the operating system and various application software installed on the computer device 10, such as a computer-readable instruction of a voice cloning method, etc. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or are to be output.

[0150] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run computer-readable instructions or process data stored in the memory 10a, such as computer-readable instructions for running the voice cloning method.

[0151] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0152] The computer device proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to speech conversion processing scenarios of different pronunciation objects. This application obtains the speech coding sequence corresponding to the source speech by inputting the source speech into the pre-trained self-supervised speech representation model; inputs the reference speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech; discretizes the speech coding sequence corresponding to the source speech to obtain the discretized speech coding; uses the discretized speech coding as the query coding in turn, and screens out the most similar coding of the target number from the speech coding sequence corresponding to the reference speech; clones the speech coding sequence according to the most similar coding of the target number corresponding to all query codings, and the discretization order of all query codings, and obtains the cloned speech coding sequence; inputs the cloned speech coding sequence into the preset vocoder to output the audio waveform. Applied to the scenarios of voice intelligent customer service and virtual voice generation, it is simpler and more cost-effective to provide personalized voice selection services for customer service or customers.

[0153] The present application also provides another implementation manner, that is, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the voice cloning method as described above.

[0154] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to speech conversion processing scenarios of different pronunciation objects. This application obtains the speech coding sequence corresponding to the source speech by inputting the source speech into the pre-trained self-supervised speech representation model; inputs the reference speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech; discretizes the speech coding sequence corresponding to the source speech to obtain the discretized speech coding; uses the discretized speech coding as the query coding in turn, and screens out the most similar coding of the target number from the speech coding sequence corresponding to the reference speech; clones the speech coding sequence according to the most similar coding of the target number corresponding to all query codings, and the discretization order of all query codings, and obtains the cloned speech coding sequence; inputs the cloned speech coding sequence into the preset vocoder to output the audio waveform. Applied to the scenarios of voice intelligent customer service and virtual voice generation, it is simpler and more cost-effective to provide personalized voice selection services for customer service or customers.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0156] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A voice cloning method, characterized in that: The steps include: Inputting a source speech into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the source speech, wherein the source speech is a speech of a first speaker; Inputting a reference speech into a pre-trained self-supervised speech representation model to obtain a speech coding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker; Discretizing the speech coding sequence corresponding to the source speech to obtain a discretized speech coding; The discretized speech codes are used as query codes in sequence, and the target number of most similar codes are selected from the speech code sequence corresponding to the reference speech; According to the most similar codes of the target number corresponding to all query codes respectively and the discretization order of all query codes, the voice code sequence is cloned to obtain the cloned voice code sequence; The cloned speech coding sequence is input into a preset vocoder to obtain an audio waveform output by the vocoder.

2. The voice cloning method according to claim 1, characterized in that: Before executing the step of inputting the source speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, the method further includes: Respectively identifying the language categories of the source speech and the reference speech; By comparison, determining whether the language category of the source speech and the reference speech is consistent; If the language categories of the source speech and the reference speech are inconsistent, the source speech is input into a preset speech translator to obtain a translated speech that is consistent with the language category of the reference speech; The translated speech is updated to the source speech.

3. The voice cloning method according to claim 1, characterized in that: Before executing the step of inputting the source speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the source speech, the method further includes: Based on the language category of the reference speech, obtaining a corpus training set of the language category, wherein the corpus training set includes batches of speech of the language category; The corpus training set is input into a preset self-supervised speech representation model to perform model pre-training to obtain a pre-trained self-supervised speech representation model.

4. The voice cloning method according to claim 3, characterized in that: The step of inputting the corpus training set into a preset self-supervised speech representation model to perform model pre-training to obtain a pre-trained self-supervised speech representation model specifically includes: Extracting the speech coding sequence of each training sample in the corpus training set by using the preset self-supervised speech representation model; Using the kmeans method, discretizing the speech coding sequence corresponding to each training sample in the speech training set; Obtaining the discretized speech code contained in each training sample in the speech training set; Performing labeling processing on all the discretized speech codes in the speech training set to obtain label values ​​corresponding to all the discretized speech codes; The corresponding association relationship between the discretized speech coding and the label value is deployed as pre-trained knowledge to the preset self-supervised speech representation model to obtain the pre-trained self-supervised speech representation model.

5. The voice cloning method according to claim 4, characterized in that: The step of inputting the reference speech into the pre-trained self-supervised speech representation model to obtain the speech coding sequence corresponding to the reference speech specifically includes: Extracting the speech coding sequence of the reference speech through the pre-trained self-supervised speech representation model; According to the corresponding association relationship between the discretized speech codes and the label values ​​in the pre-trained self-supervised speech representation model, all the discretized speech codes and label values ​​contained in the speech code sequence of the reference speech are identified.

6. The voice cloning method according to claim 4, characterized in that: The step of using the discretized speech codes as query codes in sequence and selecting the target number of most similar codes from the speech code sequence corresponding to the reference speech specifically includes: sequentially taking the discretized speech codes contained in the source speech as query codes; Identifying the label value corresponding to the current query code according to the corresponding association relationship between the discretized speech code and the label value in the pre-trained self-supervised speech representation model; A label value distance comparison algorithm is adopted to screen out the discretized speech codes with the smallest label value distance of the target number from all the discretized speech codes included in the speech code sequence of the reference speech.

7. The voice cloning method according to claim 1, characterized in that: The step of cloning the speech coding sequence according to the most similar codes of the target number corresponding to all query codes and the discretization order of all query codes to obtain the cloned speech coding sequence specifically includes: Obtain the most similar codes of the target quantity corresponding to all query codes respectively; According to the label value of the most similar code of the target number corresponding to the current query code, the majority label value of the current query code is identified; If the majority tag value identification result corresponding to the current query code is unique, then the most similar code corresponding to the majority tag value is updated to be the clone code of the current query code; Otherwise, according to the label values ​​of the most similar codes of the target number corresponding to the current query code, an average label value is calculated for the current query code, and a clone code of the current query code is determined according to the average label value; The clone codes corresponding to all query codes are obtained, and all clone codes are serialized and sorted according to the discretization order of all query codes to obtain the cloned speech code sequence.

8. A voice cloning device, characterized in that: include: A source speech serialization encoding module, used for inputting a source speech into a pre-trained self-supervised speech representation model to obtain a speech encoding sequence corresponding to the source speech, wherein the source speech is the speech of the first speaker; A reference speech serialization encoding module, used for inputting a reference speech into a pre-trained self-supervised speech representation model to obtain a speech encoding sequence corresponding to the reference speech, wherein the reference speech is the speech of the second speaker; A discretization processing module, used for discretizing the speech coding sequence corresponding to the source speech to obtain a discretized speech coding; A most similar code screening module, used to use the discretized speech codes as query codes in sequence, and screen out a target number of most similar codes from the speech code sequence corresponding to the reference speech; A speech coding sequence cloning module, used to clone the speech coding sequence according to the most similar codes of the target number corresponding to all query codes respectively and the discretization order of all query codes, and obtain the cloned speech coding sequence; The vocoder output module is used to input the cloned speech coding sequence into a preset vocoder to obtain the audio waveform output by the vocoder.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the voice cloning method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the voice cloning method according to any one of claims 1 to 7 are implemented.

Images