Sound cloning method, sound cloning apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a sound cloning method, a sound cloning device, an electronic device and a storage medium, belonging to the field of artificial intelligence and financial technology. The method comprises: obtaining sample voice data and target voice data of a target object; performing phoneme extraction processing on the target voice data to obtain target phoneme data; performing feature extraction on the target voice data to obtain target feature embedding data and speaker features; extracting target embedding data from the sample voice data according to the target feature embedding data; performing model optimization on a preset initial speech synthesis model according to the target embedding data and the target phoneme data to obtain a target speech synthesis model; and inputting preset target text data into the target speech synthesis model for voice cloning to obtain cloned voice data with speaker features. Embodiments of the present application can migrate information in the sample voice data that matches the target object, obtain a model that can clone speaker features, and improve the naturalness of cloned voices.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and financial technology, in particular to a voice cloning method, a voice cloning device, an electronic device and a storage medium. BACKGROUND

[0002] Voice cloning is a method of copying and simulating human voice using artificial intelligence technology. In the financial field, voice cloning technology is often used in intelligent customer service, voice assistants and other functions. Virtual customer service created using voice cloning technology can help customers with self-service and improve customer experience.

[0003] Currently, the main scheme of voice cloning is to train or fine-tune a synthesis model using a small amount of voice data of a target user to obtain a cloned synthesis model, input new text data into the cloned synthesis model for voice synthesis, and thus obtain cloned voice of the target user. However, this scheme has prosody problems in the voice synthesis process, resulting in synthesized voice lacking emotion and being unnatural. Therefore, how to improve the naturalness of cloned voice has become a technical problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a voice cloning method, a voice cloning device, an electronic device and a storage medium, which aims to improve the naturalness of cloned voice.

[0005] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a voice cloning method, which comprises:

[0006] obtaining sample voice data and target voice data of a target object;

[0007] performing phoneme extraction processing on the target voice data to obtain target phoneme data;

[0008] performing feature extraction on the target voice data to obtain target feature embedding data and speaker features; wherein the speaker features are used to identify the identity of the target object;

[0009] extracting target embedding data from the sample voice data according to the target feature embedding data;

[0010] performing model optimization on a preset initial voice synthesis model according to the target embedding data and the target phoneme data to obtain a target voice synthesis model;

[0011] inputting preset target text data into the target voice synthesis model for voice cloning to obtain cloned voice data with the speaker features.

[0012] In some embodiments, the model optimization is performed on the preset initial speech synthesis model according to the target embedding data and the target phoneme data to obtain a target speech synthesis model, including:

[0013] The target embedding data and the target phoneme data are input into the initial speech synthesis model to perform duration prediction to obtain predicted duration;

[0014] The predicted duration and the target phoneme data are calculated by a dynamic time warping algorithm to obtain model optimization data;

[0015] The initial speech synthesis model is optimized according to the model optimization data to obtain the target speech synthesis model.

[0016] In some embodiments, the predicted duration and the target phoneme data are calculated by a dynamic time warping algorithm to obtain model optimization data, including:

[0017] The target phoneme data is processed by spectrum synthesis according to the predicted duration to obtain predicted spectrum data;

[0018] The predicted spectrum data is calculated by the dynamic time warping algorithm to obtain the model optimization data.

[0019] In some embodiments, the target phoneme data is processed by spectrum synthesis according to the predicted duration to obtain predicted spectrum data, including:

[0020] The target phoneme data is extracted to obtain original phoneme embedding data;

[0021] The original phoneme embedding data is aligned according to the predicted duration to obtain predicted phoneme embedding data;

[0022] The predicted phoneme embedding data is decoded to obtain the predicted spectrum data.

[0023] In some embodiments, the predicted spectrum data is calculated by the dynamic time warping algorithm to obtain the model optimization data, including:

[0024] The target speech data is processed by spectrum conversion to obtain target spectrum data;

[0025] The loss data between the predicted spectrum data and the target spectrum data is calculated by the dynamic time warping algorithm to obtain the model optimization data.

[0026] In some embodiments, the extracting target embedding data from the sample voice data according to the target feature embedding data comprises:

[0027] performing sample feature extraction on the sample voice data to obtain sample feature embedding data;

[0028] performing similarity calculation on the target feature embedding data and the sample feature embedding data to obtain feature similarity data;

[0029] performing sorting on the sample feature embedding data according to the feature similarity data to obtain feature embedding sorting data;

[0030] screening out a preset number of the feature embedding sorting data to obtain the target embedding data.

[0031] In some embodiments, the performing phoneme extraction processing on the target voice data to obtain target phoneme data comprises:

[0032] performing text conversion on the target voice data to obtain preliminary text data;

[0033] performing normalization processing on the preliminary text data to obtain normalized data of the preliminary text data;

[0034] performing word segmentation processing on the normalized data to obtain dictionary data of the preliminary text data;

[0035] performing phoneme transcription processing on the dictionary data to obtain the target phoneme data.

[0036] To achieve the above object, a second aspect of the embodiment of the present application proposes a sound cloning device, which comprises:

[0037] a voice data acquisition module configured to acquire sample voice data and target voice data of a target object;

[0038] a phoneme data extraction module configured to perform phoneme extraction processing on the target voice data to obtain target phoneme data;

[0039] a feature extraction module configured to perform feature extraction on the target voice data to obtain target feature embedding data and speaker features; wherein the speaker features are used to identify the identity of the target object;

[0040] a target data acquisition module configured to extract target embedding data from the sample voice data according to the target feature embedding data;

[0041] a model optimization module configured to perform model optimization on a preset initial voice synthesis model according to the target embedding data and the target phoneme data to obtain a target voice synthesis model;

[0042] a voice cloning module, configured to input preset target text data into the target voice synthesis model to perform voice cloning, and obtain cloned voice data with the speaker feature.

[0043] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0044] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0045] The sound cloning method, sound cloning device, electronic device and storage medium provided by the present application obtain sample voice data and target voice data of a target object; perform phoneme extraction processing on the target voice data to obtain target phoneme data; perform feature extraction on the target voice data to obtain target feature embedding data and a speaker feature used to identify the identity of the target object; extract target embedding data from the sample voice data according to the target feature embedding data; perform model optimization on a preset initial voice synthesis model according to the target embedding data and the target phoneme data to obtain a target voice synthesis model; and input preset target text data into the target voice synthesis model to perform voice cloning, thereby obtaining cloned voice data with the speaker feature. The voice data matched with the target object voice data is obtained from the sample voice data, and the prosody information of the voice data is migrated to optimize the initial voice synthesis model, thereby obtaining a target voice synthesis model capable of cloning the speaker feature of the target object, and improving the naturalness of the cloned voice. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the sound cloning method provided by the embodiments of the present application;

[0047] Figure 2 is a flowchart of step S102 in Figure 1

[0048] Figure 3 is a flowchart of step S104 in Figure 1

[0049] Figure 4 is a flowchart of step S105 in Figure 1

[0050] Figure 5 is a flowchart of step S106 in Figure 4 ​​​the flowchart of step S402 in

[0051] Figure 6 is Figure 5 the flowchart of step S501 in

[0052] Figure 7 is Figure 5 the flowchart of step S502 in

[0053] Figure 8 is a structural schematic diagram of a sound cloning device provided by an embodiment of the present application;

[0054] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0056] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0058] First, several terms involved in the present application are analyzed:

[0059] Artificial intelligence (AI): is a new technical science of studying, developing the theory, method, technology and application system for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence, the research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0060] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.), NLP is a branch of artificial intelligence, and is a cross discipline of computer science and linguistics, and is often referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0061] Information extraction (Information Extraction): a text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, chapters, and text information is composed of small specific units, such as words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, names, places, etc. from text data is text information extraction, of course, the information extracted by text information extraction technology can be various types of information.

[0062] Embedding: Embedding is a vector representation that represents an object, such as a word, a product, or a movie, with a low-dimensional vector. The nature of this embedding vector is to make vectors with similar distances correspond to objects with similar meanings. The essence of embedding is a mapping from semantic space to vector space, while trying to maintain the relationship between the original samples in semantic space as much as possible in vector space, such as two semantically close words in vector space.

[0063] Decoder: Decoder is used to convert the fixed vector generated by encoding into an output sequence; where the input sequence can be text, speech, image, video, etc.; the output sequence can be text, image, etc.

[0064] Phoneme: Phoneme is the smallest unit of a syllable divided according to the natural properties of speech, analyzed according to the pronunciation action in the syllable. The phonemes of each language are different, for example, the phonemes of Chinese are the pronunciation rules of Chinese pinyin, which include initial consonants and final consonants; while the phonemes of English are composed of vowels, consonants and phonetic symbols.

[0065] Dynamic Time Warping (DTW): Dynamic Time Warping is a method for calculating the similarity between two time series, which is an algorithm for calculating similarity by stretching the time series through warping. For a given two time series, such as speech sequences, using the dynamic time warping method can align the two sequences, reduce the influence of delay and fluctuation, and better measure the similarity of the two time series.

[0066] Voice cloning is a method of copying and simulating human voice using artificial intelligence technology. In the financial field, voice cloning technology is often used in intelligent customer service, voice assistants and other functions.

[0067] Intelligent customer service: Financial institutions can use voice cloning technology to create virtual customer service representatives to help customers with self-service, account information queries and answers to common questions, etc.

[0068] Voice assistant: Voice cloning technology can be used to develop voice assistant applications to improve customer experience and reduce labor costs.

[0069] In addition, voice cloning technology can also be used for risk assessment, anti-fraud, identity verification, etc., to help financial institutions better control and manage risks and protect customer rights and interests.

[0070] Currently, the mainstream solution for voice cloning is to train or fine-tune a speech synthesis model using a small amount of speech data of the target user, and then perform speech synthesis using new text data to generate cloned voice. However, this solution has certain prosody problems and causes the synthesized speech to lack certain emotion and naturalness.

[0071] Based on this, the embodiments of the present application provide a voice cloning method, a voice cloning device, an electronic device and a storage medium, aiming to improve the naturalness of cloned voice.

[0072] The voice cloning method, the voice cloning device, the electronic device and the storage medium provided by the embodiments of the present application are specifically explained through the following embodiments. First, the voice cloning method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0075] The voice cloning method provided by the embodiments of the present application relates to the field of artificial intelligence and financial technology. The voice cloning method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN and basic cloud computing services such as big data and artificial intelligence platform; the software can be an application for implementing the voice cloning method, etc., but is not limited to the above forms.

[0076] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0077] It should be noted that in each of the specific embodiments of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data, and user location information, etc., the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0078] Figure 1 is an optional flowchart of the voice cloning method provided by the embodiments of the present application, Figure 1 The method in the method can include but is not limited to steps S101 to S106.

[0079] Step S101, obtaining sample voice data and target voice data of a target object;

[0080] Step S102, performing phoneme extraction processing on the target voice data to obtain target phoneme data;

[0081] Step S103, performing feature extraction on the target voice data to obtain target feature embedding data and speaker features; wherein the speaker features are used to identify the identity of the target object;

[0082] Step S104, extracting target embedding data from the sample voice data according to the target feature embedding data;

[0083] In step S105, the initial preset voice synthesis model is optimized according to the target embedding data and the target phoneme data, and a target voice synthesis model is obtained.

[0084] In step S106, the preset target text data is input into the target voice synthesis model for voice cloning, and cloned voice data with speaker characteristics is obtained.

[0085] The steps S101 to S106 shown in the embodiments of the present application obtain sample voice data and target voice data of a target object, perform phoneme extraction processing on the target voice data to obtain target phoneme data, perform feature extraction on the target voice data to obtain target feature embedding data and speaker characteristics for identifying the identity of the target object, extract target embedding data from the sample voice data according to the target feature embedding data, optimize the initial preset voice synthesis model according to the target embedding data and the target phoneme data to obtain a target voice synthesis model, and input the preset target text data into the target voice synthesis model for voice cloning to obtain cloned voice data with speaker characteristics. By obtaining voice data matching the target voice data from the sample voice data and migrating the prosody information of the voice data to optimize the initial voice synthesis model, a target voice synthesis model capable of cloning the timbre characteristics of the target object is obtained, thereby improving the naturalness of the cloned voice.

[0086] In step S101 of some embodiments, the sample voice data can be an open source voice data set, such as AISHELL3, Baker data, etc., or voice data collected from a preset population. The sample voice data includes multiple speakers, and each speaker records a certain amount of voice data.

[0087] It should be noted that the amount of target voice data of the target object is small, for example, 10 audios or 20 audios, which is much less than the amount of sample voice data. By finding the most similar sample voice data to the target voice data of the target object from all sample voice data, the amount of the most similar sample voice data is greater than the amount of the target voice data, so the model can be adjusted using the most similar sample voice data. The voice synthesized by the target voice synthesis model has similar timbre characteristics and prosodic characteristics to the target object, and the cloned voice is more natural.

[0088] Please refer to Figure 2 In some embodiments, step S102 can include but is not limited to steps S201 to S204:

[0089] In step S201, the target voice data is converted into text to obtain preliminary text data.

[0090] In step S202, the preliminary text data is normalized to obtain normalized data of the preliminary text data.

[0091] In step S203, the normalized data is segmented to obtain dictionary data of the preliminary text data.

[0092] In step S204, the dictionary data is subjected to phoneme transcription to obtain target phoneme data.

[0093] The steps S201 to S204 shown in the embodiments of the present application convert the target voice data into preliminary text data, and normalize, segment, and transcribe the preliminary text data to obtain target phoneme data. The normalized preliminary text data can reduce noise in the data, ensure more accurate phoneme feature embedding data extracted subsequently, improve the precision of model optimization, and thus make the cloned voice have better naturalness.

[0094] In step S202 of some embodiments, the normalization process includes regularization processing to convert the preliminary text data into a text sequence in a unified format, for example, converting the number "123456" to "one two three four five six"; converting the date "2002-01-01" to "2002-01-01"; and converting the currency "￥10" to "ten yuan".

[0095] In step S204 of some embodiments, the dictionary data is subjected to phoneme transcription to obtain phoneme data, which can be transcribed according to a pre-constructed phoneme reference table. For example, the word "clone" corresponds to the phoneme "ke4 long1" in the phoneme reference table, and when the input dictionary data contains the word "clone", the word "clone" is converted to the phoneme "ke4 long1".

[0096] In step S103 of some embodiments, the target voice data is subjected to feature extraction, including extracting target feature embedding data of the target object, the target feature embedding feature being an average calculated speaker embedding feature (Speaker-Embedding); the specific steps can include but are not limited to:

[0097] Extracting the speaker embedding feature of the target voice data to obtain temporary embedding feature;

[0098] Performing average calculation on the temporary embedding feature to obtain target feature embedding data.

[0099] In addition, the feature extraction on the target voice data further includes extracting a speaker feature of the target object, where the speaker feature includes a pitch-embedding feature, an energy-embedding feature, and a speaker-embedding feature, and is used to represent identity information of the target object.

[0100] In some embodiments, the feature extraction on the target voice data adopts a language classification model, such as Deep-Speaker, GE2E, and the like.

[0101] Please refer to Figure 3 In some embodiments, the step S104 can include, but is not limited to, steps S301 to S304.

[0102] In step S301, sample feature extraction is performed on the sample voice data to obtain sample feature embedding data.

[0103] In step S302, similarity calculation is performed on the target feature embedding data and the sample feature embedding data to obtain feature similarity data.

[0104] In step S303, the sample feature embedding data is sorted according to the feature similarity data to obtain feature embedding sorting data.

[0105] In step S304, a preset number of feature embedding sorting data are screened to obtain target embedding data.

[0106] The steps S301 to S304 shown in the embodiments of the present application obtain sample feature embedding data of sample voice data, perform similarity calculation on the target embedding data and the sample feature embedding data to obtain feature similarity data, then screen sample feature embedding data satisfying a preset condition to obtain target embedding data. The target embedding data is used to migrate prosody information, optimize an initial voice synthesis model, and finally obtain a target voice synthesis model capable of cloning voice features of a target object, thereby improving naturalness of cloned voice.

[0107] It should be noted that the sample feature extraction on the sample voice data is performed to extract a speaker-embedding feature (Speaker-Embedding) of each speaker in the sample voice data.

[0108] In step S303 of some embodiments, the sample feature embedding data is sorted according to the feature similarity data. The sorting manner can be reverse sorting or positive sorting, which is not limited in the present embodiment.

[0109] In step S304 of some embodiments, a preset number of feature embedding ranking data are screened out, where the preset number can be one or multiple, which is not limited in the present embodiment.

[0110] In some embodiments, one embedding data with the maximum similarity data is screened out from the feature embedding ranking data to obtain target embedding data, which represents the speaker feature embedding data most similar to the target feature embedding data of the target object. The initial speech synthesis model is subsequently optimized using the feature embedding data.

[0111] Referring to Figure 4 In some embodiments, step S105 can include but is not limited to steps S401 to S403:

[0112] In step S401, the target embedding data and the target phoneme data are input into the initial speech synthesis model for duration prediction to obtain predicted duration.

[0113] In step S402, a dynamic time warping algorithm is used to perform time warping calculation on the predicted duration and the target phoneme data to obtain model optimization data.

[0114] In step S403, the initial speech synthesis model is optimized according to the model optimization data to obtain a target speech synthesis model.

[0115] The steps S401 to S403 shown in the present embodiment optimize the initial speech synthesis model through the model optimization data to finally obtain the target speech synthesis model, so that the target speech synthesis model is optimized according to the acoustic characteristics of the target object, and the timbre and prosody of the synthesized speech conform to the acoustic characteristics of the target object, so that the cloned voice has better naturalness.

[0116] It should be noted that the preset initial speech synthesis model is a model obtained by pre-training sample speech data through a preset open-source speech synthesis model. The preset open-source speech model can be Fastspeech2, Transformer TTS, Tacotron2, etc., which is not limited in the present embodiment.

[0117] In some embodiments, the step of obtaining the initial speech synthesis model can include but is not limited to the following steps:

[0118] Performing prosody accuracy detection on the sample speech data to obtain qualified speech data;

[0119] Inputting the qualified speech data into a preset speech synthesis model for pre-training to obtain the initial speech synthesis model.

[0120] It should be noted that the preset voice synthesis model is an open source synthesis model, which is Fastspeech2 in the embodiment, and can also be other models, such as Transformer TTS, Tacotron2, etc.

[0121] In step S401 of some embodiments, step S401 can include but is not limited to including the following steps:

[0122] The target embedding data and the target phoneme data are input into the initial voice synthesis model.

[0123] The duration of the target phoneme data is predicted according to the target embedding data and the preset corpus, and the predicted duration is obtained.

[0124] It should be noted that the predicted duration is predicted according to the speaker embedding feature most similar to the target object, which can be understood as the predicted duration closest to the phoneme duration when the target object speaks, and can be used to represent the phoneme duration of the target object, so as to learn the speaking feature of the target object from a small amount of voice data of the target object, thereby improving the naturalness of the cloned voice.

[0125] Please refer to Figure 5 In some embodiments, step S402 can include but is not limited to steps S501 to S502:

[0126] Step S501, performing spectral synthesis processing on the target phoneme data according to the predicted duration to obtain predicted spectral data;

[0127] Step S502, performing loss calculation on the predicted spectral data using a dynamic time warping algorithm to obtain model optimization data.

[0128] The steps S501 to S502 shown in the embodiment of the application use a dynamic warping algorithm to calculate the difference between the predicted spectral data and the actual data, obtain loss data as model optimization data, and use the loss data to adjust the initial voice synthesis model, so that the timbre and prosody of the synthesized voice meet the acoustic characteristics of the target object, and the cloned voice has better naturalness.

[0129] The predicted spectral data in the embodiment is mel-spectral data.

[0130] It should be noted that the predicted spectral data can also be Fourier spectral data, and can also be other spectral data, which can be set according to the use scenario.

[0131] Please refer to Figure 6 In some embodiments, step S501 can include but is not limited to steps S601 to S603:

[0132] Step S601, phoneme feature extraction is performed on the target phoneme data to obtain original phoneme embedding data;

[0133] Step S602, the original phoneme embedding data is aligned according to the predicted duration to obtain predicted phoneme embedding data;

[0134] Step S603, the predicted phoneme embedding data is decoded to obtain predicted spectrum data.

[0135] The steps S601 to S603 shown in the embodiments of the present application, by performing phoneme feature extraction on the target phoneme data, obtain original phoneme embedding data, wherein the original phoneme embedding features include time features, frequency features, etc. of the phoneme; the time features of the original phoneme embedding data are aligned according to the predicted duration to obtain predicted phoneme embedding data; the predicted phoneme embedding data is decoded to obtain predicted spectrum data. Through the alignment processing, the original phoneme embedding features have the linguistic features of the target object, and the subsequent predicted spectrum data is used to adjust the initial speech synthesis model, so that the timbre and prosody of the synthesized speech conform to the acoustic features of the target object, so that the cloned voice has better naturalness.

[0136] Please refer to Figure 7 In some embodiments, step S502 can further include but is not limited to steps S701 to S702:

[0137] Step S701, performing spectrum conversion processing on the target speech data to obtain target spectrum data;

[0138] Step S702, using a dynamic time warping algorithm to calculate the loss data between the predicted spectrum data and the target spectrum data to obtain model optimization data.

[0139] The steps S701 to S702 shown in the embodiments of the present application, by performing spectrum conversion processing on the target speech data, obtain target spectrum data, which is actual data; then using a dynamic warping algorithm to calculate the loss value between the predicted spectrum data and the target spectrum data, obtain model optimization data. The initial speech synthesis model is adjusted by the model optimization data, so that the timbre and prosody of the synthesized speech conform to the acoustic features of the target object, so that the cloned voice has better naturalness.

[0140] In step S701 of some embodiments, the target speech data is spectrum converted by a decoder.

[0141] In step S702 of some embodiments, the model optimization data includes but is not limited to: spectral loss data, duration data. Specifically, the spectral loss data is the data obtained by calculating the loss data between the predicted spectral data and the target spectral data in step S702 using the dynamic time warping algorithm; the duration loss data is the predicted duration obtained in step S401.

[0142] It should be noted that when the dynamic time warping algorithm is used, the gradient descent method is added for parameter adjustment, which overcomes the problem of process dispersion and non-differentiability when calculating the loss value between the predicted spectral data and the target spectral data.

[0143] In step S106 of some embodiments, the target speech synthesis model obtained after model optimization can realize cloning of the acoustic characteristics of the target object. By inputting the preset text data into the target speech synthesis model, cloned speech data with the speaker characteristics of the target object can be obtained. The timbre and prosody of the synthesized speech of the target speech synthesis model obtained after model optimization conform to the acoustic characteristics of the target object, so that the cloned voice has better naturalness.

[0144] In an embodiment, using the voice cloning method in intelligent customer service can include but is not limited to the following steps:

[0145] Obtaining sample speech data and target speech data of a target object; wherein the target object is an object that the intelligent customer service needs to imitate, and permission of the target object is required before using the target speech data of the target object;

[0146] Performing prosodic accuracy detection on the sample speech data to obtain qualified speech data;

[0147] Inputting the qualified speech data into a preset speech synthesis model for pre-training to obtain an initial speech synthesis model.

[0148] Performing text conversion on the target speech data to obtain preliminary text data;

[0149] Performing normalization processing on the preliminary text data to obtain normalized data of the preliminary text data;

[0150] Performing word segmentation processing on the normalized data to obtain dictionary data of the preliminary text data;

[0151] Performing phoneme transcription processing on the dictionary data to obtain target phoneme data;

[0152] Performing feature extraction on the target speech data to obtain target feature embedding data and speaker characteristics; wherein the speaker characteristics are used to identify the identity of the target object;

[0153] Sample feature extraction is performed on the sample speech data to obtain sample feature embedding data;

[0154] Similarity calculation is performed on the target feature embedding data and the sample feature embedding data to obtain feature similarity data;

[0155] The sample feature embedding data is sorted according to the feature similarity data to obtain feature embedding sorting data;

[0156] A preset number of feature embedding sorting data are screened out to obtain target embedding data;

[0157] The target embedding data and the target phoneme data are input into the initial speech synthesis model for duration prediction to obtain predicted duration;

[0158] Phoneme feature extraction is performed on the target phoneme data to obtain original phoneme embedding data;

[0159] The original phoneme embedding data is aligned according to the predicted duration to obtain predicted phoneme embedding data;

[0160] The predicted phoneme embedding data is decoded to obtain predicted spectrum data;

[0161] The target speech data is subjected to spectrum conversion processing to obtain target spectrum data;

[0162] The dynamic time warping algorithm is used to calculate the loss data between the predicted spectrum data and the target spectrum data to obtain model optimization data;

[0163] The initial speech synthesis model is optimized according to the model optimization data to obtain a target speech synthesis model;

[0164] The preset target text data is input into the target speech synthesis model for speech cloning to obtain cloned speech data with speaker characteristics.

[0165] Finally, through training and optimization of the initial speech synthesis model, a target speech synthesis model capable of cloning the speaking characteristics of the target object is obtained, and the target speech synthesis model is used in the intelligent customer service function to create a virtual customer service representative using voice cloning technology, so that the speech synthesized by the virtual customer service representative has the emotions and naturalness of the target object, and can help customers with self-service, account information query and answering common questions, etc.

[0166] In addition, the voice cloning method can also be used for the development of voice assistant functions, which can improve the naturalness of synthesized speech, improve the user experience of customers, and also reduce labor costs. The specific model training and optimization method is basically the same as the specific embodiments of the voice cloning method described above, and will not be repeated here.

[0167] Please refer to Figure 8 The embodiment of the application further provides a sound cloning device which can realize the sound cloning method, and the device comprises:

[0168] The speech data acquisition module 801 is configured to acquire sample speech data and target speech data of a target object.

[0169] The phoneme data extraction module 802 is configured to perform phoneme extraction processing on the target speech data to obtain target phoneme data.

[0170] The feature extraction module 803 is configured to perform feature extraction on the target speech data to obtain target feature embedding data and speaker features; wherein the speaker features are used to identify the identity of the target object.

[0171] The target data acquisition module 804 is configured to extract target embedding data from the sample speech data according to the target feature embedding data.

[0172] The model optimization module 805 is configured to perform model optimization on a preset initial speech synthesis model according to the target embedding data and the target phoneme data to obtain a target speech synthesis model.

[0173] The speech cloning module 806 is configured to input preset target text data to the target speech synthesis model to perform speech cloning and obtain cloned speech data with speaker features.

[0174] In some embodiments of the phoneme data extraction module 802, the phoneme data extraction module 802 further comprises:

[0175] The text conversion sub-module is configured to perform text conversion on the target speech data to obtain preliminary text data.

[0176] The normalization sub-module is configured to perform normalization processing on the preliminary text data to obtain normalized data of the preliminary text data.

[0177] The word segmentation sub-module is configured to perform word segmentation processing on the normalized data to obtain dictionary data of the preliminary text data.

[0178] The phoneme transcription sub-module is configured to perform phoneme transcription processing on the dictionary data to obtain the target phoneme data.

[0179] In some embodiments of the target data acquisition module 804, the target data acquisition module 804 further comprises:

[0180] The sample feature extraction sub-module is configured to perform sample feature extraction on the sample speech data to obtain sample feature embedding data.

[0181] The similarity calculation submodule is configured to calculate the similarity between the target feature embedding data and the sample feature embedding data to obtain feature similarity data.

[0182] The sorting submodule is configured to sort the sample feature embedding data according to the feature similarity data to obtain feature embedding sorting data.

[0183] The screening submodule is configured to screen out a preset number of feature embedding sorting data to obtain target embedding data.

[0184] In the model optimization module 805 of some embodiments, the model optimization module 805 further includes:

[0185] The duration prediction submodule is configured to input the target embedding data and the target phoneme data into the initial speech synthesis model to perform duration prediction to obtain predicted duration.

[0186] The warping submodule is configured to perform time warping calculation on the predicted duration and the target phoneme data by using a dynamic time warping algorithm to obtain model optimization data.

[0187] The model optimization submodule is configured to optimize the initial speech synthesis model according to the model optimization data to obtain a target speech synthesis model.

[0188] In the warping submodule of some embodiments, the warping submodule further includes:

[0189] The spectrum synthesis unit is configured to perform spectrum synthesis processing on the target phoneme data according to the predicted duration to obtain predicted spectrum data.

[0190] The loss calculation unit is configured to perform loss calculation on the predicted spectrum data by using a dynamic time warping algorithm to obtain model optimization data.

[0191] In the spectrum synthesis unit of some embodiments, the spectrum synthesis unit further includes:

[0192] The phoneme feature extraction subunit is configured to perform phoneme feature extraction on the target phoneme data to obtain original phoneme embedding data.

[0193] The alignment subunit is configured to perform alignment processing on the original phoneme embedding data according to the predicted duration to obtain predicted phoneme embedding data.

[0194] The decoding subunit is configured to perform decoding processing on the predicted phoneme embedding data to obtain predicted spectrum data.

[0195] In the loss calculation unit of some embodiments, the loss calculation unit further includes:

[0196] The spectrum conversion sub-unit is configured to perform spectrum conversion processing on the target voice data to obtain target spectrum data.

[0197] The calculation sub-unit is configured to calculate loss data between the predicted spectrum data and the target spectrum data by using a dynamic time warping algorithm to obtain model optimization data.

[0198] The specific implementation of the sound cloning device is basically the same as the specific implementation of the sound cloning method described above, and will not be repeated here.

[0199] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the sound cloning method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0200] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0201] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0202] The memory 902 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the sound cloning method of the embodiments of the present application.

[0203] The input / output interface 903 is configured to realize information input and output.

[0204] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0205] A bus 905 transmits information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0206] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other within the device through the bus 905.

[0207] The computer program stored in the computer readable storage medium is executed by the processor to implement the sound cloning method.

[0208] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0209] The sound cloning method, the sound cloning device, the electronic device, and the storage medium provided by the present application obtain sample voice data and target voice data of a target object; perform phoneme extraction processing on the target voice data to obtain target phoneme data; perform feature extraction on the target voice data to obtain target feature embedding data and speaker features for identifying the identity of the target object; extract target embedding data from the sample voice data according to the target feature embedding data; perform model optimization on a preset initial voice synthesis model according to the target embedding data and the target phoneme data to obtain a target voice synthesis model; and input preset target text data into the target voice synthesis model to perform voice cloning to obtain cloned voice data with speaker features. By obtaining voice data matching the target object voice data from the sample voice data and migrating the prosody information of the voice data to optimize the initial voice synthesis model, a target voice synthesis model capable of cloning the voice timbre features of the target object is obtained, thereby improving the naturalness of the cloned voice.

[0210] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0211] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0212] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0213] Those skilled in the art can understand that all or some steps in the above disclosed method, functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0214] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those clearly listed steps or units, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0215] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0216] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0217] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0218] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0219] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0220] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of sound cloning, the method comprising: The method comprises: acquiring sample voice data and target voice data of a target object; performing phoneme extraction processing on the target voice data to obtain target phoneme data; performing feature extraction on the target voice data to obtain target feature embedding data and speaker features; wherein the target feature embedding data is used to represent speaker embedding features of the target object, and the speaker features are used to identify the identity of the target object; extracting target embedding data from the sample voice data according to the target feature embedding data; wherein the target embedding data is used to represent speaker feature embedding data similar to the target feature embedding data of the target object; performing model optimization on a preset initial voice synthesis model according to the target embedding data and the target phoneme data to obtain a target voice synthesis model; inputting preset target text data into the target voice synthesis model for voice cloning to obtain cloned voice data with the speaker features.

2. The voice cloning method of claim 1, wherein, The method comprises: inputting the target embedding data and the target phoneme data into the initial voice synthesis model for duration prediction to obtain predicted duration; performing time warping calculation on the predicted duration and the target phoneme data by using a dynamic time warping algorithm to obtain model optimization data; optimizing the initial voice synthesis model according to the model optimization data to obtain the target voice synthesis model.

3. The sound cloning method of claim 2, wherein, The method comprises: performing spectral synthesis processing on the target phoneme data according to the predicted duration to obtain predicted spectral data; performing loss calculation on the predicted spectral data by using the dynamic time warping algorithm to obtain the model optimization data.

4. The voice cloning method of claim 3, wherein, The method comprises: performing phoneme feature extraction on the target phoneme data to obtain original phoneme embedding data; performing alignment processing on the original phoneme embedding data according to the predicted duration to obtain predicted phoneme embedding data; performing decoding processing on the predicted phoneme embedding data to obtain the predicted spectral data.

5. The voice cloning method of claim 3, wherein, The method comprises: performing spectral conversion processing on the target voice data to obtain target spectral data; calculating loss data between the predicted spectral data and the target spectral data by using the dynamic time warping algorithm to obtain the model optimization data.

6. The sound cloning method according to any one of claims 1-5, wherein, The method comprises: performing sample feature extraction on the sample voice data to obtain sample feature embedding data; performing similarity calculation on the target feature embedding data and the sample feature embedding data to obtain feature similarity data; According to the feature similarity data, the sample feature embedding data is sorted to obtain feature embedding sorting data; A preset number of the feature embedding sorting data is screened to obtain the target embedding data.

7. The sound cloning method according to any one of claims 1-5, wherein, The phoneme extraction processing on the target speech data is performed to obtain target phoneme data, including: Text conversion is performed on the target speech data to obtain preliminary text data; Normalization processing is performed on the preliminary text data to obtain normalized data of the preliminary text data; Word segmentation processing is performed on the normalized data to obtain dictionary data of the preliminary text data; Phoneme transcription processing is performed on the dictionary data to obtain the target phoneme data.

8. A sound cloning apparatus, characterized by comprising: The device comprises: A speech data acquisition module is configured to acquire sample speech data and target speech data of a target object; A phoneme data extraction module is configured to perform phoneme extraction processing on the target speech data to obtain target phoneme data; A feature extraction module is configured to perform feature extraction on the target speech data to obtain target feature embedding data and speaker features; wherein the target feature embedding data is used to represent speaker embedding features of the target object, and the speaker features are used to identify the identity of the target object; A target data acquisition module is configured to extract target embedding data from the sample speech data according to the target feature embedding data; wherein the target embedding data is used to represent speaker feature embedding data similar to the target feature embedding data of the target object; A model optimization module is configured to perform model optimization on a preset initial speech synthesis model according to the target embedding data and the target phoneme data to obtain a target speech synthesis model; A voice cloning module is configured to input preset target text data to the target speech synthesis model for voice cloning to obtain cloned voice data with the speaker features.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the voice cloning method of any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the voice cloning method of any one of claims 1-7.

Images