Clone sound generation method, clone sound application method and clone sound generation device

Abstract

The invention provides a clone sound generation method, and relates to the technical field of voice processing, in particular to the technical field of sound cloning. According to the specific implementation scheme, in response to a clone sound synthesis request sent by an application client, a clone sound identifier and to-be-synthesized content carried by the clone sound synthesis request are obtained, and a corresponding sound clone model is determined according to the clone sound identifier; obtaining clone sound synthesis data corresponding to the to-be-synthesized content based on the to-be-synthesized content and the sound clone model, and returning the clone sound synthesis data to the application client, the application client being configured to receive and apply the clone sound synthesis data; the sound clone model is obtained according to the following mode: taking a sound sample of a user collected and sent by a collection client as training data, inputting the training data into an initial sound clone model for training, and obtaining a sound clone model corresponding to the sound sample of the user, the sound clone model is used for outputting clone sound synthesis data similar to the tone of the sound sample of the user.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to the field of sound cloning technology. Background Art

[0002] Currently, voice cloning is used in various scenarios, such as live broadcast, social media, voice navigation, etc. Among them, some users have the need to customize cloned voices, but the current process of customizing cloned voices is complicated and requires technicians to record and clone the user's voice. It is difficult for users to generate the required cloned voices by themselves. Summary of the invention

[0003] The present disclosure provides a cloned sound generation method, a cloned sound application method and a device for solving at least one of the above-mentioned technical problems.

[0004] According to a first aspect of the present disclosure, a method for generating a cloned sound is provided, wherein the method is applied to a server and comprises:

[0005] In response to a cloned sound synthesis request sent by an application client, obtaining a cloned sound identifier and content to be synthesized carried in the cloned sound synthesis request, and determining a corresponding sound cloning model according to the cloned sound identifier;

[0006] Obtaining cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and returning the cloned sound synthesis data to the application client, wherein the application client is configured to receive and apply the cloned sound synthesis data;

[0007] The sound cloning model is obtained in the following manner:

[0008] The user's voice sample collected and sent by the collection client is used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

[0009] According to a second aspect of the present disclosure, a method for applying a cloned sound is provided, wherein the method is applied to an application client and comprises:

[0010] A cloned sound synthesis request is generated based on the content to be synthesized and the cloned sound identifier of the sound cloning model corresponding to the required cloned sound, and sent to the server, wherein the server is configured to, in response to applying the cloned sound synthesis request, obtain the cloned sound identifier and the content to be synthesized carried by the cloned sound synthesis request, determine the corresponding sound cloning model according to the cloned sound identifier, obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data;

[0011] The cloned sound synthesis data returned by the server is received, and the cloned sound synthesis data is applied.

[0012] According to a third aspect of the present disclosure, a method for generating a cloned sound is provided, wherein the method is applied to a collection client, and the method includes:

[0013] Collect user's voice samples and send them to the server;

[0014] Wherein, the server processes the sound sample according to the above method.

[0015] According to a fourth aspect of the present disclosure, a device for generating cloned sounds is provided, wherein the device comprises:

[0016] A model determination module, configured to respond to a cloned sound synthesis request sent by an application client, obtain a cloned sound identifier and content to be synthesized carried in the cloned sound synthesis request, and determine a corresponding sound cloning model according to the cloned sound identifier;

[0017] a sound synthesis module, configured to obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data to the application client, wherein the application client is configured to receive and apply the cloned sound synthesis data;

[0018] The device further comprises: a model acquisition module, configured to obtain the sound cloning model in the following manner:

[0019] The user's voice sample collected and sent by the collection client is used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

[0020] According to a fifth aspect of the present disclosure, a sound cloning application device is provided, wherein the device comprises:

[0021] A request module, configured to generate a cloned sound synthesis request based on the content to be synthesized and the cloned sound identifier of the sound cloning model corresponding to the required cloned sound, and send the request to a server, wherein the server is configured to, in response to applying the cloned sound synthesis request, obtain the cloned sound identifier and the content to be synthesized carried in the cloned sound synthesis request, determine the corresponding sound cloning model according to the cloned sound identifier, obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data;

[0022] The application module is used to receive the cloned sound synthesis data returned by the server and apply the cloned sound synthesis data.

[0023] According to a sixth aspect of the present disclosure, a collection module is provided, which is used to collect a user's voice sample and send it to a server;

[0024] Wherein, the server processes the sound sample according to the above method.

[0025] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0029] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0030] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0031] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0033] Figure 1 is a flowchart of a method for generating cloned sounds provided by the first embodiment of the present disclosure;

[0034] Figure 2 is a flowchart of a method for obtaining a sound cloning model provided in the first embodiment of the present disclosure;

[0035] Figure 3 is a flow chart of a sound cloning application method provided by the second embodiment of the present disclosure;

[0036] Figure 4 is a flowchart of a method for generating cloned sounds provided in the third embodiment of the present disclosure;

[0037] Figure 5 is a flowchart of another method for generating cloned sounds provided in the third embodiment of the present disclosure;

[0038] Figure 6 is a schematic structural diagram of a cloned sound generation device provided by a fourth embodiment of the present disclosure;

[0039] Figure 7 is a structural schematic diagram of a sound cloning application device provided by the fifth embodiment of the present disclosure;

[0040] Figure 8 is a structural schematic diagram of a cloned sound generation device provided by a sixth embodiment of the present disclosure;

[0041] Fig. 9 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0044] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0045] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0046] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0047] The cloned sound generation method and the cloned sound application method according to the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the cloned sound generation method and the cloned sound application method provided by the present disclosure can be executed by a server.

[0048] The method provided by the present disclosure can be applied in a three-end interaction scenario, where the three ends include: a server, an application client, and a collection client. The collection client is used to collect user voice samples; the server is used to train a voice cloning model based on the voice samples to perform speech synthesis; the application client is the client for the final application and playback of the cloned voice.

[0049] In some scenarios, the collection client is a client using a microservice architecture, in other words, a mini-program client; the application client is a client using a global wide area network, in other words, a web client. In some cases, the sound receiving devices of web clients vary greatly, and it is difficult to collect sound using a web mechanism, while the sound receiving devices of mobile devices are relatively uniform, the sound receiving environment is good, and the coding of the mini-program is easier to implement the sound collection function. Therefore, in the present disclosure, a mini-program is used as a collection client to collect sound samples, which are input to the server for training to obtain a sound cloning model. The web client as an application client directly communicates with the server and requests the server to perform cloned sound synthesis, thereby combining the characteristics of different clients to achieve convenient and accurate cloned sound generation and application.

[0050] In the first embodiment disclosed, see Figure 1 , Figure 1 A schematic flow chart of a method for generating a cloned sound provided by the first embodiment of the present disclosure is shown. The method can be applied to a server. The method includes:

[0051] S101 . In response to a cloned sound synthesis request sent by an application client, obtain a cloned sound identifier and content to be synthesized carried in the cloned sound synthesis request, and determine a corresponding sound cloning model according to the cloned sound identifier.

[0052] S102: Obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data to the application client.

[0053] The application client is configured to receive and apply the cloned sound synthesis data.

[0054] Among them, see Figure 2 , Figure 2 A flow chart of a method for obtaining a sound cloning model is shown. The sound cloning model is obtained according to the following method:

[0055] S100: The user's voice samples collected and sent by the collection client are used as training data and input into an initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice samples.

[0056] The voice cloning model is used to output cloned sound synthesis data that is close to the timbre of the user's voice sample.

[0057] The present disclosure provides a method through the above-mentioned method, so that users can clone sounds by themselves and obtain cloned sounds. The whole process does not require other personnel, and the acquisition of cloned sounds is automated. In addition, the sound cloning and application are realized in a three-end interactive manner. The acquisition client collects sound samples and transmits them to the server for training to obtain a sound cloning model of the sound samples. On this basis, when the cloned sound needs to be applied, the application client requests the server to call the sound cloning model to generate cloned sound data corresponding to the content to be synthesized, and returns it to the application client. The application client can apply the cloned sound in various scenarios. In this way, the collection, training and application of sounds are integrated, making sound cloning more flexible.

[0058] The sound cloning model or the initial sound cloning model can be implemented in a variety of ways, for example, a large model, a recurrent neural network (RNN), an encoder (Transformer) and other network structures, which are not limited here.

[0059] In some examples, the clone identifier is obtained in the following manner:

[0060] S100', after obtaining the trained sound cloning model, generate a cloned sound identifier for the sound cloning model, and send the cloned sound identifier and the mapping relationship between the cloned sound identifier and the sound sample to the application client.

[0061] That is, the server stores the mapping relationship between the cloned sound identifier (mid)-sound cloning model-sound sample. After training the sound cloning model corresponding to a sound sample, the server generates a cloned sound identifier for the sound cloning model, and then sends the cloned sound identifier and the mapping relationship to the application client. The application client saves the cloned sound identifier and the mapping relationship. When the sound cloning model of a sound sample is to be used in the application, the cloned sound identifier is configured and sent to the server. The server calls the corresponding sound cloning model according to the cloned sound identifier to use the sound cloning model to output the cloned sound that expresses the content to be synthesized.

[0062] In some examples, S100 includes:

[0063] S1001. The user's voice samples collected and sent by the acquisition client are used as training data and input into the initial voice cloning model for training.

[0064] S1002. During the training process, the sound sample is identified to obtain the sound adjustment parameters, and the model parameters of the initial sound cloning model are adjusted according to the sound adjustment parameters to obtain a trained sound cloning model corresponding to the user's sound sample.

[0065] Among them, the sound adjustment parameters can represent sound features, including at least one of the following: keyword parameters of the sound sample, emotion feature parameters of the sound sample, accent feature parameters of the sound sample, and of course, other sound adjustment parameters can also be used, which are not limited here.

[0066] Among them, the keyword parameters of the sound sample can be obtained by converting the sound sample into text and extracting the keywords therein as keyword parameters; the emotion feature parameters can be obtained by identifying the emotion features according to the sound sample, and using the identified emotion features as emotion feature parameters, wherein the identified emotion features can be obtained by an emotion feature extraction model; the accent feature parameters can be obtained by identifying the accent features according to the sound sample, and using the identified accent features as accent feature parameters, wherein the identified accent features can be obtained by an accent feature extraction model. During the training process, the sound adjustment parameters can be extracted according to the user's sound sample, and then the model parameters can be automatically adjusted according to the sound adjustment parameters so that the cloned sound output by the sound cloning model adapts to the user's sound features, thereby improving the fidelity of the generated cloned sound.

[0067] In some examples, S100 further includes:

[0068] S1001. The user's voice samples collected and sent by the acquisition client are used as training data and input into the initial voice cloning model for training.

[0069] S1002. During the training process, the sound sample is identified to obtain the sound adjustment parameters, and the model parameters of the initial sound cloning model are adjusted according to the sound adjustment parameters to obtain a trained sound cloning model corresponding to the user's sound sample.

[0070] S1003. Extract any sound segment of the sound sample to obtain a test content text.

[0071] S1004: Input the test content text into the sound cloning model to obtain output cloned sound synthesis data corresponding to the test content text.

[0072] S1005. Perform similarity verification based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text. If the verification passes, determine that the sound cloning model is compliant.

[0073] Among them, a piece of text can be extracted as the test content text based on the sound sample, and then input into the sound cloning model, and the output cloned sound synthesis data of the test content text read in the cloned sound can be obtained through the Text To Speech (TTS) technology, and then the similarity between the cloned sound synthesis data of the test content text and the corresponding reason (i.e., the sound sample) of the test content text is compared to perform similarity compliance verification to ensure the accuracy of the sound cloning model and the cloning quality.

[0074] For example, in some embodiments, the content of the sound sample is "It is xx:00 on xx / xx / xx, and I am recording a sound sample." The sound paragraph of "I am recording a sound sample" is intercepted and converted into a test content text. The test content text of "I am recording a sound sample" is then input into the sound cloning model corresponding to the trained sound sample. The output is obtained through TTS technology, and the cloned sound synthesis data of "I am recording a sound sample" is read aloud in the cloned sound of the sound sample. Then, the similarity between the sound paragraph of the sound sample and the obtained cloned sound synthesis data is compared. When the similarity is higher than a preset threshold, the sound cloning model is determined to be compliant.

[0075] In some examples, the similarity comparison may be in multiple dimensions, including, for example, timbre similarity of voices, pronunciation similarity of keywords, intonation similarity, emotion similarity, etc., which are not limited here. In other words, in S1005, the similarity qualification check is performed based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text, including at least one of the following:

[0076] Based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text, the timbre similarity qualification verification is performed.

[0077] Based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text, the pronunciation similarity of the keyword is verified.

[0078] Based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text, the intonation similarity qualification verification is performed.

[0079] Based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text, the emotion similarity qualification verification is performed.

[0080] Finally, the similarity scores of multiple similarity checks are combined to determine the final similarity check score. The average of the similarity scores of multiple similarity checks can be used as the final score, or different weights can be set for each similarity check, and then a weighted calculation is performed, and the result of the weighted calculation is used as the final score. There is no limitation here.

[0081] In some examples, the content to be synthesized may be in the form of speech or text. In the case of the content to be synthesized in the form of text, the text-based content to be synthesized is directly input into the sound cloning model, and the output cloned sound synthesis data of the content to be synthesized in the cloned sound is obtained through the TTS technology.

[0082] In the case where the content to be synthesized is in the form of speech, obtaining cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model in S102 includes:

[0083] Step 1: Convert the content to be synthesized in speech form into content to be synthesized in text form by voice-to-text conversion.

[0084] Step 2: Input the content to be synthesized in text form into the sound cloning model, and obtain the cloned sound synthesis data corresponding to the content to be synthesized through text-to-speech conversion.

[0085] In this case, the server first converts the speech into text and then inputs it into the sound cloning model. For the application client used by the user, the user directly inputs the original sound to obtain a cloned sound with the same content, which improves the user experience.

[0086] In some examples, the cloned sound synthesis data is in the form of streaming data or audio packets. That is, the present disclosure supports two synthesis methods.

[0087] In the first mode, the cloned sound synthesis data is streamed. In this case, the user's application client sends a cloned sound synthesis request including a cloned sound identifier and the content to be synthesized to the server in real time. The server calls the sound cloning model corresponding to the cloned sound identifier, inputs the content to be synthesized for cloned sound synthesis, outputs multiple sub-cloned sound synthesis data in segments, and sends them to the application client in real time. Among them, multiple sub-cloned sound synthesis data constitute a complete cloned sound synthesis data. In this mode, due to the high real-time requirements for the synthesized cloned sound, the streaming sub-cloned sound synthesis data is returned to the application client in a streaming manner, that is, the completed synthesis task is split into multiple sub-tasks. After completing the cloned sound synthesis of a sub-task, the sub-cloned sound synthesis data obtained by this sub-task is returned to the application client, so that the application client side can realize the cloned sound application of synthesis and playback to meet the real-time conversation scenario.

[0088] In the second method, the cloned sound synthesis data is in the form of an audio package. In this case, the user's application client generates a cloned sound synthesis request based on the content to be synthesized and the cloned sound identifier and sends it to the server through the interface. The server calls the sound cloning model corresponding to the cloned sound identifier, completely synthesizes the content to be synthesized into the cloned sound synthesis data, and then packages the cloned sound synthesis data into an audio package, and sends the audio package to the application client. The application client parses the audio package to obtain the cloned sound synthesis data.

[0089] The above two forms can be switched and combined arbitrarily to increase the flexibility of scene application.

[0090] In some examples, S102 specifically includes:

[0091] Step 1: Obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model.

[0092] Step 2: Obtain sound effect parameters according to the content to be synthesized, and adjust the cloned sound synthesis data based on the sound effect parameters.

[0093] Step 3: Return the cloned sound synthesis data to the application client.

[0094] The sound effect parameters are parameters that can add sound effects to the cloned sound, and matching sound effect parameters can be obtained according to the emotional characteristics, keywords, etc. of the content to be synthesized. As an example, the sound effect parameters include at least one of the following: sound effect parameters for adjusting intonation; sound effect parameters for adjusting accent; sound effect parameters for adjusting the sound hearing space; sound effect parameters for adjusting the sound age; sound effect parameters for adjusting the sound gender, etc., which are not limited here.

[0095] For example, in some embodiments, the sound effect parameters are sound effect parameters for adjusting the age of the sound; based on step two, the content to be synthesized is analyzed, and based on the analysis results, it is determined that the age of the sound is relatively young, then sound effect parameters for reducing the age of the sound are generated, and the cloned sound synthesis data are adjusted according to the sound effect parameters, and then the adjusted cloned sound synthesis data with reduced sound age is sent to the client. In this way, the generated cloned sound can be adjusted intelligently.

[0096] In disclosing the second embodiment, see Figure 3 , Figure 3 A flow chart of a sound cloning application method provided by the second embodiment of the present disclosure is shown. The method can be applied to an application client. The method includes:

[0097] S201: Generate a clone sound synthesis request based on the content to be synthesized and the clone sound identifier of the sound cloning model corresponding to the required clone sound, and send the request to the server.

[0098] Among them, the server is configured to respond to an application clone sound synthesis request, obtain the clone sound identifier and the content to be synthesized carried in the clone sound synthesis request, determine the corresponding sound cloning model according to the clone sound identifier, obtain the clone sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the clone sound synthesis data.

[0099] S202: Receive the cloned sound synthesis data returned by the server, and apply the cloned sound synthesis data.

[0100] In some examples, the application client is a client for live broadcasting; the application of cloned sound synthesis data in S202 includes:

[0101] The cloned sound synthesis data is converted into live streaming data, and the live streaming data is applied in the live broadcast.

[0102] In some cases, the cloned sound synthesis data output by the sound cloning model can be applied in live broadcast. In this case, the script can be directly configured on the live broadcast application client. The script contains the cloned sound identification and the content to be synthesized. When running the script, the server is requested to obtain the cloned sound synthesis data. The application client then converts the cloned sound synthesis data into live broadcast stream data and applies it in the live broadcast, thereby achieving the effect of real-time application of sound cloning for digital human live broadcast in the live broadcast.

[0103] In some examples, before S201, the method further includes:

[0104] S200: Receive and save the cloned sound identifier sent by the server and the mapping relationship between the cloned sound identifier and the sound sample.

[0105] The cloned sound identifier is generated according to the method of S100' in the first disclosed embodiment.

[0106] The server stores the mapping relationship between cloned sound identifier (mid)-sound cloning model-sound sample. After training the sound cloning model corresponding to a sound sample, the server generates a cloned sound identifier for the sound cloning model, and then sends the cloned sound identifier and the mapping relationship to the application client. The application client saves the cloned sound identifier and the mapping relationship. When the sound cloning model of a sound sample is to be used in the application, the cloned sound identifier is configured and sent to the server. The server calls the corresponding sound cloning model according to the cloned sound identifier to use the sound cloning model to output the cloned sound that expresses the content to be synthesized.

[0107] In disclosing the second embodiment, see Figure 4 , Figure 4 A schematic flow chart of a method for generating a cloned sound provided in the third embodiment of the present disclosure is shown. The method can be applied to a collection client. The method includes:

[0108] S301: Collect the user's voice sample and send it to the server.

[0109] The server processes the sound sample according to the method of the first disclosed embodiment.

[0110] In some scenarios, such as when the application client is a web client, since the web client has poor sound collection performance, the mobile device's audio equipment can be used to collect sound, separating the collection and application steps. The sound samples collected by the collection client are sent to the server for training.

[0111] For some examples, see Figure 5 , Figure 5 A flow chart of another method for generating cloned sounds provided in the third embodiment of the disclosure is shown, which includes:

[0112] S401: Perform noise detection on the user's voice collection environment.

[0113] S402: Generate a user voice collection prompt based on the noise detection result.

[0114] Noise detection is performed to ensure the quality of the collection environment. Sound collection prompts may include various forms, such as text or voice prompting the user that the sound environment is poor, text or voice prompting the user to start collection, etc., which are not limited here.

[0115] S403: According to a preset standard text, collect a voiceprint sample of the user reading the standard text.

[0116] S404: Collect other sound samples of the user according to a preset time period.

[0117] Among them, the sound samples include: voiceprint samples and other sound samples. Among them, the voiceprint samples are used as reference data for model training, that is, the standard voice of the user; other sound samples are used to increase the comprehensiveness of the sound samples, provide sufficient and diverse samples as training data, and ensure the training effect of the model. Therefore, the voiceprint samples of each user can use standard texts with fixed content to ensure the consistency of the content of the voiceprint samples of each user; other sound samples can be freely read by the user, or provide optional texts, and they can be collected for a long enough time, so the preset time is set to collect enough samples.

[0118] Among them, S403 and S404 are an implementation method of collecting the user's voice sample in S301.

[0119] S405: Perform quality processing on the sound sample to obtain a processed sound sample.

[0120] The quality processing includes at least one of the following: noise reduction processing, volume standardization processing, and text alignment processing. Through quality processing, the quality of the sound sample is guaranteed to improve the quality of model training.

[0121] S406: Send the sound sample to the server.

[0122] Based on Figure 1 The same principle, Figure 6 A cloned sound generating device 60 provided in a fourth embodiment of the present disclosure is shown, and the device comprises:

[0123] The model determination module 601 is used to respond to the cloned sound synthesis request sent by the application client, obtain the cloned sound identification and the content to be synthesized carried in the cloned sound synthesis request, and determine the corresponding sound cloning model according to the cloned sound identification;

[0124] The sound synthesis module 602 is used to obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data to the application client, wherein the application client is configured to receive and apply the cloned sound synthesis data;

[0125] The device further includes: a model acquisition module 600, which is used to obtain the sound cloning model according to the following method:

[0126] The user's voice samples collected and sent by the collection client are used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

[0127] In some examples, the apparatus further includes: an identifier generation module, configured to obtain a cloned sound identifier in the following manner:

[0128] After the trained sound cloning model is obtained, a cloned sound identifier is generated for the sound cloning model, and the cloned sound identifier and the mapping relationship between the cloned sound identifier and the sound sample are sent to the application client.

[0129] In some examples, the model acquisition module is specifically used to:

[0130] The user's voice samples collected and sent by the acquisition client are used as training data and input into the initial voice cloning model for training;

[0131] During the training process, the sound sample is identified to obtain the sound adjustment parameters, and the model parameters of the initial sound cloning model are adjusted according to the sound adjustment parameters to obtain a trained sound cloning model corresponding to the user's sound sample.

[0132] In some examples, the sound adjustment parameter includes at least one of the following: a keyword parameter of the sound sample, an emotion characteristic parameter of the sound sample, and an accent characteristic parameter of the sound sample.

[0133] In some examples, the model acquisition module is also used to:

[0134] Extract any sound segment of the sound sample to obtain the test content text;

[0135] Input the test content text into the sound cloning model to obtain output cloned sound synthesis data corresponding to the test content text;

[0136] A similarity verification is performed based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text. If the verification passes, it is determined that the sound cloning model is compliant.

[0137] In some examples, the content to be synthesized is content to be synthesized in the form of speech;

[0138] The sound synthesis module is specifically used for:

[0139] By converting speech to text, the content to be synthesized in speech form is converted into content to be synthesized in text form;

[0140] The content to be synthesized in text form is input into the sound cloning model, and the cloned sound synthesis data corresponding to the content to be synthesized is obtained through text-to-speech conversion.

[0141] In some examples, the cloned sound synthesis data is in the form of streaming data or audio packets.

[0142] In some examples, the apparatus further comprises:

[0143] The effect adjustment module is used to obtain sound effect parameters according to the content to be synthesized, and adjust the cloned sound synthesis data based on the sound effect parameters.

[0144] Based on Figure 3 The same principle, Figure 7 A sound cloning application device 70 provided in the fifth embodiment of the present disclosure is shown, and the device includes:

[0145] The request module 701 is used to generate a cloned sound synthesis request based on the cloned sound identifier of the sound cloning model corresponding to the content to be synthesized and the required cloned sound, and send it to the server, wherein the server is configured to respond to the application cloned sound synthesis request, obtain the cloned sound identifier and the content to be synthesized carried in the cloned sound synthesis request, determine the corresponding sound cloning model according to the cloned sound identifier, obtain the cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data.

[0146] The application module 702 is used to receive the cloned sound synthesis data returned by the server and apply the cloned sound synthesis data.

[0147] In some examples, the apparatus further comprises:

[0148] An identification saving module, used for receiving and saving the cloned sound identification sent by the server and the mapping relationship between the cloned sound identification and the sound sample;

[0149] The cloned sound identifier is generated in the manner of S100 ′ according to the disclosed embodiment.

[0150] Based on Figure 4 The same principle, Figure 8 A sixth embodiment of the present disclosure provides a cloned sound generation device 80, which includes:

[0151] The collection module 801 is used to collect the user's voice samples and send them to the server;

[0152] The server processes the sound sample according to the method of the first disclosed embodiment.

[0153] In some examples, the apparatus further includes:

[0154] Environmental detection module, used for:

[0155] Perform noise detection on the user's sound collection environment;

[0156] Based on the noise detection results, a user voice collection prompt is generated.

[0157] In some examples, the acquisition module is specifically used to:

[0158] Voiceprint samples and other sound samples;

[0159] Collect user voice samples, including:

[0160] According to the preset standard text, collect the voiceprint sample of the user reading the standard text;

[0161] Collect other sound samples of the user according to the preset duration.

[0162] In some examples, the apparatus further comprises:

[0163] The sound processing module is used to: perform quality processing on the sound sample to obtain the processed sound sample;

[0164] The quality processing includes at least one of the following: noise reduction processing, volume standardization processing, and text alignment processing.

[0165] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle.

[0167] The computer instructions of the non-transitory computer-readable storage medium storing computer instructions are used to make the computer execute the above method.

[0168] A computer program product comprises a computer program, which implements the above method when executed by a processor.

[0169] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0170] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0171] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0172] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the cloned sound generation method and the cloned sound application method. For example, in some embodiments, the cloned sound generation method and the cloned sound application method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the cloned sound generation method and the cloned sound application method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured as the cloned sound generation method and the cloned sound application method in any other appropriate manner (for example, by means of firmware).

[0173] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0175] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0180] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a cloned sound, wherein: Applied on the server side, the method includes: In response to a cloned sound synthesis request sent by an application client, obtaining a cloned sound identifier and content to be synthesized carried in the cloned sound synthesis request, and determining a corresponding sound cloning model according to the cloned sound identifier; Obtaining cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and returning the cloned sound synthesis data to the application client, wherein the application client is configured to receive and apply the cloned sound synthesis data; The sound cloning model is obtained in the following manner: The user's voice sample collected and sent by the collection client is used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

2. The method according to claim 1, wherein: The cloned sound identifier is obtained in the following manner: After the trained sound cloning model is obtained, a cloned sound identifier is generated for the sound cloning model, and the cloned sound identifier and the mapping relationship between the cloned sound identifier and the sound sample are sent to the application client.

3. The method according to claim 1 or 2, wherein: The method uses the user's voice sample collected and sent by the collection client as training data, inputs it into the initial voice cloning model for training, and obtains a trained voice cloning model corresponding to the user's voice sample, including: The user's voice samples collected and sent by the acquisition client are used as training data and input into the initial voice cloning model for training; During the training process, the voice sample is identified to obtain a voice adjustment parameter, and the model parameters of the initial voice cloning model are adjusted according to the voice adjustment parameter to obtain a trained voice cloning model corresponding to the voice sample of the user.

4. The method according to claim 3, wherein: The sound adjustment parameter includes at least one of the following: a keyword parameter of the sound sample, an emotion feature parameter of the sound sample, and an accent feature parameter of the sound sample.

5. The method according to claim 3 or 4, wherein: The method further comprises: identifying the sound sample to obtain the sound associated parameters, adjusting the model parameters of the initial sound cloning model according to the sound associated parameters, and obtaining the trained sound cloning model corresponding to the sound sample of the user; and then using the sound sample of the user collected and sent by the collection client as training data, inputting the sound sample into the initial sound cloning model for training, and obtaining the trained sound cloning model corresponding to the sound sample of the user. Extract any sound segment of the sound sample to obtain a test content text; Inputting the test content text into the sound cloning model to obtain output cloned sound synthesis data corresponding to the test content text; A similarity verification is performed based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text. If the verification passes, it is determined that the sound cloning model is compliant.

6. The method according to any one of claims 1 to 5, wherein: The content to be synthesized is content to be synthesized in the form of speech; The step of obtaining cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model includes: By converting speech to text, the content to be synthesized in speech form is converted into content to be synthesized in text form; The content to be synthesized in text form is input into the sound cloning model, and cloned sound synthesis data corresponding to the content to be synthesized is obtained through text-to-speech conversion.

7. The method according to any one of claims 1 to 6, wherein: The cloned sound synthesis data is in the form of streaming data or audio packets.

8. The method according to any one of claims 1 to 7, wherein: After obtaining the cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and before returning the cloned sound synthesis data to the application client, the method further includes: A sound effect parameter is obtained according to the content to be synthesized, and the cloned sound synthesis data is adjusted based on the sound effect parameter.

9. The method according to any one of claims 1 to 8, wherein: The collection client is a client that adopts a microservice architecture; the application client is a client that adopts a global wide area network.

10. A method for cloning sound, wherein: Applied in an application client, the method includes: A cloned sound synthesis request is generated based on the content to be synthesized and the cloned sound identifier of the sound cloning model corresponding to the required cloned sound, and sent to the server, wherein the server is configured to, in response to applying the cloned sound synthesis request, obtain the cloned sound identifier and the content to be synthesized carried by the cloned sound synthesis request, determine the corresponding sound cloning model according to the cloned sound identifier, obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data; receiving the cloned sound synthesis data returned by the server, and applying the cloned sound synthesis data; The sound cloning model is obtained in the following manner: The user's voice sample collected and sent by the collection client is used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

11. The method according to claim 10, wherein: Before generating a clone sound synthesis request based on the content to be synthesized and the clone sound identifier of the sound clone model corresponding to the required clone sound and sending the request to the server, the method further includes: Receiving and saving the cloned sound identifier sent by the server and the mapping relationship between the cloned sound identifier and the sound sample; Wherein, the cloned sound identifier is generated according to the method described in claim 2.

12. The method according to claim 10 or 11, wherein: The application client is a client for live broadcast; The applying the cloned sound to synthesize data comprises: The cloned sound synthesis data is converted into live streaming data, and the live streaming data is applied in the live broadcast.

13. A method for generating a cloned sound, wherein: Applied in a collection client, the method includes: Collect user's voice samples and send them to the server; Wherein, the server processes the sound sample according to any one of the methods according to claims 1-9.

14. The method according to claim 13, wherein: Before collecting the user's voice sample and sending it to the server, the method further includes: Performing noise detection on the user's voice collection environment; Based on the noise detection result, a user voice collection prompt is generated.

15. The method according to claim 13 or 14, wherein: The sound samples include: voiceprint samples and other sound samples; The collecting of the user's voice sample comprises: According to a preset standard text, a voiceprint sample of a user reading the standard text is collected; Collect other sound samples of the user according to the preset duration.

16. The method according to any one of claims 13 to 15, wherein: After collecting the user's voice sample and before sending it to the server, the method further includes: Performing quality processing on the sound sample to obtain the processed sound sample; The quality processing includes at least one of the following: noise reduction processing, volume standardization processing, and text alignment processing.

17. A device for generating cloned sound, wherein: The device includes: A model determination module, configured to respond to a cloned sound synthesis request sent by an application client, obtain a cloned sound identifier and content to be synthesized carried in the cloned sound synthesis request, and determine a corresponding sound cloning model according to the cloned sound identifier; a sound synthesis module, configured to obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data to the application client, wherein the application client is configured to receive and apply the cloned sound synthesis data; The device further comprises: a model acquisition module, configured to obtain the sound cloning model in the following manner: The user's voice sample collected and sent by the collection client is used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

18. The device according to claim 17, wherein: The device further comprises: an identifier generating module, configured to obtain the cloned sound identifier in the following manner: After the trained sound cloning model is obtained, a cloned sound identifier is generated for the sound cloning model, and the cloned sound identifier and the mapping relationship between the cloned sound identifier and the sound sample are sent to the application client.

19. The device according to claim 17 or 18, wherein: The model acquisition module is specifically used for: The user's voice samples collected and sent by the acquisition client are used as training data and input into the initial voice cloning model for training; During the training process, the voice sample is identified to obtain a voice adjustment parameter, and the model parameters of the initial voice cloning model are adjusted according to the voice adjustment parameter to obtain a trained voice cloning model corresponding to the voice sample of the user.

20. The device according to claim 19, wherein The sound adjustment parameter includes at least one of the following: a keyword parameter of the sound sample, an emotion feature parameter of the sound sample, and an accent feature parameter of the sound sample.

21. The device according to claim 19 or 20, wherein: The model acquisition module is also used for: Extract any sound segment of the sound sample to obtain a test content text; Inputting the test content text into the sound cloning model to obtain output cloned sound synthesis data corresponding to the test content text; A similarity verification is performed based on the cloned sound synthesis data corresponding to the test content text and the sound sample corresponding to the test content text. If the verification passes, it is determined that the sound cloning model is compliant.

22. The device according to any one of claims 17 to 21, wherein: The content to be synthesized is content to be synthesized in the form of speech; The sound synthesis module is specifically used for: By converting speech to text, the content to be synthesized in speech form is converted into content to be synthesized in text form; The content to be synthesized in text form is input into the sound cloning model, and cloned sound synthesis data corresponding to the content to be synthesized is obtained through text-to-speech conversion.

23. The device according to any one of claims 17 to 22, wherein: The cloned sound synthesis data is in the form of streaming data or audio packets.

24. The device according to any one of claims 17 to 23, wherein: The device also includes: The effect adjustment module is used to obtain sound effect parameters according to the content to be synthesized, and adjust the cloned sound synthesis data based on the sound effect parameters.

25. A sound cloning application device, wherein: The device includes: A request module, configured to generate a cloned sound synthesis request based on the content to be synthesized and the cloned sound identifier of the sound cloning model corresponding to the required cloned sound, and send the request to a server, wherein the server is configured to, in response to applying the cloned sound synthesis request, obtain the cloned sound identifier and the content to be synthesized carried in the cloned sound synthesis request, determine the corresponding sound cloning model according to the cloned sound identifier, obtain cloned sound synthesis data corresponding to the content to be synthesized based on the content to be synthesized and the sound cloning model, and return the cloned sound synthesis data; An application module, configured to receive the cloned sound synthesis data returned by the server, and apply the cloned sound synthesis data; The sound cloning model is obtained in the following manner: The user's voice sample collected and sent by the collection client is used as training data and input into the initial voice cloning model for training to obtain a trained voice cloning model corresponding to the user's voice sample, wherein the voice cloning model is used to output cloned sound synthesis data with a timbre similar to that of the user's voice sample.

26. The device according to claim 25, wherein The device also includes: An identification storage module, used for receiving and storing the cloned sound identification sent by the server and the mapping relationship between the cloned sound identification and the sound sample; Wherein, the cloned sound identifier is generated according to the method described in claim 2.

27. A cloned sound generating device, wherein: The device includes: The collection module is used to collect user voice samples and send them to the server; Wherein, the server processes the sound sample according to any one of the methods according to claims 1-9.

28. The device according to claim 27, wherein The device also includes: Environmental detection module, used for: Performing noise detection on the user's voice collection environment; Based on the noise detection result, a user voice collection prompt is generated.

29. The device according to claim 27 or 28, wherein The acquisition module is specifically used for: Voiceprint samples and other sound samples; The collecting of the user's voice sample comprises: According to a preset standard text, a voiceprint sample of a user reading the standard text is collected; Collect other sound samples of the user according to the preset duration.

30. The device according to any one of claims 25 to 19, wherein: The device also includes: The sound processing module is used to: perform quality processing on the sound sample to obtain the processed sound sample; The quality processing includes at least one of the following: noise reduction processing, volume standardization processing, and text alignment processing.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-9, and / or the method of any one of claims 10-12, and / or the method of any one of claims 13-16.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9, and / or the method according to any one of claims 10-12, and / or the method according to any one of claims 13-16.

33. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9, and / or the method according to any one of claims 10 to 12, and / or the method according to any one of claims 13 to 16.

Images