Method for generating voice clone model and related device

Abstract

The invention provides a method for generating a voice cloning model and a related device, and the method is applied to the field of voice cloning, and comprises the steps: obtaining a scoring result of a plurality of reference audios inputted by a user through a terminal device; training based on the plurality of reference audios and the scoring result of the user on the plurality of reference audios to obtain an acoustic feedback unit, the acoustic feedback unit being used for measuring the auditory feeling of the user on different audios; acquiring a first voice data set input by a user through a terminal device; and based on the first voice data set and the acoustic feedback unit, training the voice clone model to obtain the voice clone model. In the voice clone model training process, the preference degree of the user for different audios is considered, the scoring results of the user for the multiple reference audios are added into the voice clone model training process, the voice clone model obtained through training better meets the use requirements of the user, and the user experience is improved. And the voice generated by using the voice cloning model obtained by training can better meet the auditory feeling of the user.

Description

Technical Field

[0001] The present application relates to the field of voice cloning, and in particular to a method for generating a voice cloning model and a related device. Background Art

[0002] In recent years, with the rapid development of industries such as virtual humans, audiobooks, and video creation, more and more repetitive dubbing tasks have been replaced by synthetic voices. Voice cloning, as a speech synthesis technology that clones the timbre, rhythm, and style of the target speaker, meets the needs of speech synthesis.

[0003] Current voice cloning systems usually extract dozens or hundreds of sentences of recording data required for cloning training. The cloning engine learns the pronunciation style, rhythm, timbre and other information of the target speaker from the provided recording data. The timbre and speaking style of the cloned voice are basically consistent with the timbre and speaking style of the speaker in the recording, but the cloned voice often does not satisfy the user's auditory experience. Summary of the invention

[0004] The present application provides a method for generating a voice cloning model and a related device. The voice generated by the method for generating a voice cloning model described in the present application can better meet the user's auditory perception and improve the user's experience.

[0005] In a first aspect, the present application provides a method for generating a voice cloning model, comprising: obtaining scoring results of multiple reference audios input by a user through a terminal device; training based on the multiple reference audios and the scoring results of the multiple reference audios by the user to obtain an acoustic feedback unit, wherein the acoustic feedback unit is used to measure the user's auditory perception of different audios; obtaining a first voice data set input by the user through the terminal device; and training a voice cloning model based on the first voice data set and the acoustic feedback unit to obtain a trained voice cloning model.

[0006] During the training process of the voice cloning model, user needs and preferences are taken into consideration, and the user's scoring results for multiple reference audios are added to the training process of the voice cloning model. The trained voice cloning model can better meet user needs; when the trained voice cloning model is applied to speech synthesis service scenarios, it can better meet the user's auditory perception.

[0007] Based on the first aspect, in a possible implementation manner, before obtaining the scoring results of the multiple reference audios input by the user through the terminal device, the method further includes: obtaining feedback information input by the user through the terminal device, the feedback information including one or more of an application scenario of the voice cloning model, an emotion category used by the voice cloning model, and a language generated by the voice cloning model; screening and obtaining the multiple reference audios from an audio library according to the feedback information; and sending the multiple reference audios to the terminal device.

[0008] Furthermore, the multiple reference audios are obtained by screening according to the feedback information input by the user. Therefore, based on the feedback information input by the user and the user's scoring results for the multiple reference audios, the trained speech cloning model can better meet the user's needs, and the speech generated using the trained speech cloning model can better meet the user's auditory experience.

[0009] Based on the first aspect, in a possible implementation, the training of the voice cloning model based on the first voice data set and the acoustic feedback unit includes multiple rounds of iterative training.

[0010] In the current round of iterative training:

[0011] The speech cloning model generates optimized speech;

[0012] The optimized speech is input into the acoustic feedback unit, and the acoustic feedback unit scores the optimized speech to obtain a scoring result of the optimized speech, and the scoring result of the optimized speech is used as an input of the speech cloning model in the next round of iterative training to influence the speech cloning model to generate the next round of optimized speech;

[0013] Wherein, in the first round of iterative training, the optimized speech is generated by the speech cloning model according to the first speech data set.

[0014] The process of training the voice cloning model is the process of strengthening the learning of the acoustic feedback unit and the voice cloning model. The acoustic feedback unit is trained based on the user's feedback information and the user's scoring results for multiple reference audios. The acoustic feedback unit can reflect and embody the user's preference for different audios. Therefore, the acoustic feedback unit can be used as a basis for judging whether the optimized voice generated by the voice cloning unit meets the user's auditory perception. That is, the acoustic feedback unit can be used to score the optimized voice generated by the voice cloning unit, and the scoring result is used as the input of the voice cloning unit to influence the voice cloning unit to generate a new round of optimized voice.

[0015] Based on the first aspect, in a possible implementation, in the current round of iterative training, the scoring result of the optimized speech is used as a parameter in the loss function of the speech cloning model in the next round of iterative training, so as to affect the loss function in the speech cloning model.

[0016] Based on the first aspect, in a possible implementation, the voice cloning model is applied to any one or more of an audio book scenario, a virtual human field, and a video creation field.

[0017] Based on the first aspect, in a possible implementation, after obtaining the trained voice cloning model, the method further includes:

[0018] The server receives target information input by the user, wherein the target information includes text;

[0019] The target information is input into the trained voice cloning model to generate a second voice.

[0020] Based on the first aspect, in a possible implementation, the target information exists in the form of any one or more combinations of documents, pictures and slides.

[0021] Based on the first aspect, in a possible implementation method, the scoring results of the multiple reference audios include scoring results of each reference audio in the multiple reference audios in multiple dimensions, and the multiple dimensions include two or more dimensions of timbre, speech rhythm, pronunciation method and articulation method.

[0022] In a second aspect, the present application provides a device for generating a voice cloning model, comprising:

[0023] An acquisition module, used to acquire the scoring results of multiple reference audios input by the user through the terminal device;

[0024] an acoustic feedback module, configured to perform training based on the multiple reference audios and the scoring results of the multiple reference audios by the user to obtain an acoustic feedback unit, wherein the acoustic feedback unit is configured to measure the user's auditory perception of different audios;

[0025] The acquisition module is used to acquire a first voice data set input by the user through the terminal device;

[0026] The speech cloning module is used to train the speech cloning model based on the first speech data set and the acoustic feedback unit to obtain a trained speech cloning model.

[0027] Based on the second aspect, in a possible implementation,

[0028] The acquisition module is further used to acquire feedback information input by the user through the terminal device, wherein the feedback information includes one or more of an application scenario of the voice cloning model, an emotion category used by the voice cloning model, and a language generated by the voice cloning model;

[0029] A screening module, used for screening and obtaining the plurality of reference audios from an audio library according to the feedback information;

[0030] A sending module is used to send the multiple reference audios to the terminal device.

[0031] Based on the second aspect, in a possible implementation, the training of the voice cloning model based on the first voice data set and the acoustic feedback unit includes multiple rounds of iterative training.

[0032] In the current round of iterative training:

[0033] The voice cloning module is used to generate optimized voice;

[0034] The acoustic feedback module is used to input the optimized speech into the acoustic feedback unit, and the acoustic feedback unit scores the optimized speech to obtain a scoring result of the optimized speech. The scoring result of the optimized speech is used as an input of the speech cloning model in the next round of iterative training to influence the speech cloning model to generate the next round of optimized speech;

[0035] Wherein, in the first round of iterative training, the optimized speech is generated by the speech cloning model according to the first speech data set.

[0036] Based on the second aspect, in a possible implementation, in the current round of iterative training, the scoring result of the optimized speech is used as a parameter in the loss function of the speech cloning model in the next round of iterative training, so as to affect the loss function in the speech cloning model.

[0037] Based on the second aspect, in a possible implementation, the voice cloning model is applied to any one or more of an audio book scenario, a virtual human field, and a video creation field.

[0038] Based on the second aspect, in a possible implementation,

[0039] The acquisition module is also used to receive target information input by the user, wherein the target information includes text;

[0040] The voice cloning module is also used to input the target information into the trained voice cloning model to generate a second voice.

[0041] Based on the second aspect, in a possible implementation, the target information exists in the form of any one or more combinations of documents, pictures and slides.

[0042] Based on the second aspect, in a possible implementation method, the scoring results of the multiple reference audios include scoring results of each reference audio in the multiple reference audios in multiple dimensions, and the multiple dimensions include two or more dimensions of timbre, speech rhythm, pronunciation method and articulation method.

[0043] Each functional module of the second aspect is used to implement the method described in the first aspect and any possible implementation method of the first aspect.

[0044] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, wherein the at least one computing device comprises a memory and a processor, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect and any possible implementation method of the first aspect.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, comprising program instructions. When the program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect and any possible implementation of the first aspect.

[0046] In a fifth aspect, the present application provides a computer program product comprising instructions. When a computing device cluster runs the instructions, the computing device cluster executes the method described in the first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a system architecture provided for this application;

[0048] Figure 2 A flowchart of a method for generating a voice cloning model provided in the present application;

[0049] Figure 3 A schematic diagram of a method for training a voice cloning model provided in this application;

[0050] Figure 4 A flowchart of a training method for an acoustic feedback unit provided in this application;

[0051] Figure 5 An example diagram of feedback information provided for this application;

[0052] Figure 6An example diagram of multiple reference audios and corresponding scoring criteria provided by this application;

[0053] Figure 7 Another example diagram of multiple reference audios and corresponding scoring criteria provided by the present application;

[0054] Figure 8 A schematic diagram of reinforcement learning training for a voice cloning model provided in this application;

[0055] Fig. 9 A flowchart of an application method of a voice cloning model provided in this application;

[0056] Fig.10 A schematic diagram of the structure of a device for generating a voice cloning model provided by the present application;

[0057] Fig.11 A schematic diagram of the structure of a computing device provided for this application;

[0058] Fig.12 A schematic diagram of the structure of a computing device cluster provided for this application;

[0059] Fig.13 A schematic diagram of the structure of another computing device cluster provided in this application. DETAILED DESCRIPTION

[0060] To facilitate description and understanding of the scheme, in this application, "first", "second", etc. are used to distinguish the same thing, and "first", "second", etc. are not used for specific references; " / " represents an or relationship, for example, A / B means A or B.

[0061] This application provides a system, see Figure 1 , Figure 1 A schematic diagram of a system architecture provided in the present application, the system includes a terminal device 110, a network device 120 and at least one server 130.

[0062] The terminal device 110, for example, can be any one of electronic products such as a notebook, a desktop computer, a tablet computer, and a wearable device. The terminal device 110 can also be other electronic devices, such as an intelligent robot. The user can input feedback information and a first voice data set through the terminal device 110, and the terminal device 110 is used to receive the feedback information and the first voice data set input by the user. The terminal device 110 is also used to send the feedback information and the first voice data set input by the user to at least one server 130 through the network device 120. For the relevant introduction of the feedback information and the first voice data set, refer to the description in the method embodiment below, which will not be described in detail here.

[0063] The network device 120 is used for transmitting data between the terminal device 110 and the server 130 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. In the present application, the network device 120 is used to receive feedback information and a first voice data set sent by the terminal device 110, and send the feedback information and the first voice data set to at least one server 130.

[0064] The server 130 may be a computing device located in a cloud, wherein the cloud may be a private cloud, a public cloud, or a hybrid cloud, and the cloud may include one or more servers 130. In the present application, the server 130 is used to receive feedback information and a first voice data set sent by the terminal device 110, and is also used to perform a series of processing according to the feedback information and the first voice data set, and finally obtain a voice cloning model. For the voice cloning model, please refer to the description of the method embodiment below, which will not be introduced here.

[0065] The obtained voice cloning model can be used to process speech synthesis services (text to speech, TTS, or “text to speech” services).

[0066] The following introduces the application scenarios where the voice cloning model is applicable.

[0067] The embodiments of the present application can be applied to audiobook scenarios. For example, a user can input information from a newspaper or magazine or paper novel or electronic novel into a server through a terminal device, and the server processes the information on the newspaper or magazine or paper novel or electronic novel to obtain voice, and sends the voice to the terminal device. After receiving the voice, the terminal device broadcasts the voice, so that the user does not need to manually flip through the newspaper or magazine or paper novel or electronic novel, and does not need to use vision to see the content on the newspaper or magazine or paper novel or electronic novel, but can "read" the newspaper or magazine or paper novel or electronic novel through hearing, freeing the user's hands and bringing benefits to the user.

[0068] The embodiments of the present application can also be applied to the field of virtual humans. For example, virtual humans can be digital humans with virtual images, or intelligent robots, etc. For example, digital humans or robots can be used to provide services to users. In the process of digital humans or robots providing services to users, digital humans or robots can use voices synthesized by servers that are biased towards the user's preferences to provide services to users. For another example, a user inputs information from a newspaper or magazine or a paper novel or an electronic novel into a digital human or robot, and the digital human or robot plays the content for the user.

[0069] The embodiments of the present application can also be applied to the field of video creation. For example, when creating a video, a user needs to add voice to the video screen, and the voice is used to assist in understanding the video content. The user inputs the voice to be added into the server in the form of text through the terminal device, and the server converts the text into a voice that meets the user's preferences, and sends the voice that meets the user's preferences to the terminal device. The user adds the voice to the video, so that the video screen is played while the voice that meets the user's preferences is played.

[0070] based on Figure 1 The system shown in the present application provides a method for generating a voice cloning model. Figure 2 and Figure 3 , Figure 2 A flowchart of a method for generating a voice cloning model provided in this application is shown in FIG. Figure 3 A schematic diagram of a training method for a voice cloning model provided in this application is shown below in combination with Figure 2 and Figure 3 The present application provides a method for generating a voice cloning model, which includes but is not limited to the following description.

[0071] S101. The server trains an acoustic feedback unit based on the user's scoring results for multiple reference audios to obtain a trained acoustic feedback unit.

[0072] In one implementation, the user can input feedback information to the server through the terminal device, and the server performs training based on the feedback information input by the user and the user's scoring results for multiple reference audios to obtain a trained acoustic feedback unit. The following describes the specific implementation method of this implementation. Figure 4 , Figure 4 A flowchart of a training method for an acoustic feedback unit provided in the present application, the method includes but is not limited to the description of the contents in the following S1011 to S1016.

[0073] S1011. The server receives feedback information input by the user through a terminal device.

[0074] The feedback information input by the user includes one or more of the application scenarios of the voice cloning model, the emotion categories used by the voice cloning model, and the language generated by the voice cloning model. The application scenarios of the voice cloning model include news scenarios, live broadcast scenarios, story scenarios, education scenarios, dialogue scenarios, general scenarios, etc. The news scenario refers to the scenario of broadcasting news, that is, the scenario for broadcasting news generated by the server later; the live broadcast scenario refers to the subsequent generated voice cloning model used in the live broadcast scenario; the story scenario refers to the subsequent generated voice cloning model used in the storytelling scenario; the education scenario refers to the subsequent generated voice cloning model used in education and teaching-related scenarios; the dialogue scenario refers to the subsequent generated voice cloning model used in the character dialogue scenario; the general scenario refers to the subsequent generated voice cloning model can be used in a variety of scenarios. Application scenarios can also include other scenarios, which are not limited in this application.

[0075] The emotion category of the voice cloning model refers to the emotion category used by the voice cloning model when broadcasting the voice, and the emotion categories include happy, angry, serious, sad, surprised, and normal emotions. For example, in a news scenario, you can choose to use a serious emotion to broadcast the voice; in a live broadcast scenario, you can choose to use a happy emotion to broadcast the voice; and so on. The emotion category can also include other emotions or moods, which are not limited in this application.

[0076] The language generated by the voice cloning model refers to the language used when the voice cloning model is used to broadcast the voice, for example, it can include Mandarin, dialects and foreign languages, where dialects can include dialects from multiple places, and foreign languages ​​can include languages ​​from multiple countries. Users can select the language used by the voice cloning model based on actual application scenarios and application requirements.

[0077] The feedback information input by the user may also include other information, which is not specifically limited in this application.

[0078] See also Figure 5 , Figure 5 An example diagram of feedback information provided for this application. Figure 5 In the feedback information, the application scenarios of the voice cloning model, the emotion categories used by the voice cloning model, and the language generated by the voice cloning model are included. The application scenarios include news scenarios, live broadcast scenarios, story scenarios, education scenarios, dialogue scenarios, and general scenarios. The emotion categories include happy, angry, serious, sad, surprised, and normal emotions. The languages ​​include Mandarin, dialects, and foreign languages. Users can select the corresponding application scenarios, emotion categories, and languages ​​according to actual application requirements. Optionally, users can select one or more application scenarios, one or more emotion categories, and one or more languages ​​according to actual application requirements.

[0079] The user inputs feedback information through the terminal device, and the terminal device sends the feedback information to the server. Correspondingly, the server receives the feedback information sent by the user.

[0080] S1012: The server selects a plurality of reference audios from the audio library based on the feedback information input by the user.

[0081] The server is provided with an audio library, which includes audios of various application scenarios, various emotion types and various language versions. The audios in the audio library are collected from the voices of different people, and are synthesized from the voices of different people. The audio includes speech.

[0082] The server selects multiple reference audios that meet the user's conditions from the audio library according to the feedback information input by the user. The user conditions include the application scenario, emotion category and language selected by the user in the feedback information. It can be seen that the multiple reference audios are selected according to the user's usage requirements.

[0083] S1013: The server sends the multiple reference audios to the terminal device. Correspondingly, the terminal device receives the multiple reference audios sent by the server.

[0084] S1014: The user scores the multiple reference audios to obtain scoring results of the multiple reference audios.

[0085] After the terminal device receives the multiple reference audios filtered according to the user's feedback information sent by the server, the user plays the multiple reference audios on the terminal device and scores the multiple reference audios respectively through auditory perception to obtain the scoring results of each of the multiple reference audios.

[0086] In one implementation, the scoring criteria for each reference audio include comfortable, acceptable, average, difficult to accept, and unacceptable, and the user scores each reference audio according to the auditory experience of each reference audio, and the scoring result is one of the scoring criteria. In another implementation, the scoring criteria for each reference audio include 1 point, 2 points, 3 points, 4 points, and 5 points, and the scoring result of each reference audio is one of the scoring criteria.

[0087] See also Figure 6 , Figure 6 This is an example diagram of multiple reference audios and corresponding scoring criteria provided by this application. Figure 6 In the method, multiple reference audios include wav1, wav2, wav3...wavn, where n is a positive integer. For each reference audio, the scoring criteria include comfortable, acceptable, general, difficult to accept and unacceptable. Users can score each reference audio based on their auditory perception and obtain the scoring results of each reference audio.

[0088] In another implementation, the scoring criteria for each reference audio may include multiple dimensions. For example, for a reference audio, the scoring criteria include two or more dimensions of timbre, voice rhythm, pronunciation, and articulation. The user scores each dimension in the scoring criteria to obtain a scoring result for the reference audio, wherein the scoring result includes scores for multiple dimensions. Figure 7 The example diagram shown, Figure 7 It includes multiple reference audios, and the scoring criteria for each reference audio include four dimensions: timbre, speech rhythm, pronunciation method and articulation method. For a reference audio, the user can score the four dimensions of timbre, speech rhythm, pronunciation method and articulation method based on the auditory experience of the reference audio to obtain a scoring result for the reference audio.

[0089] In another implementation, for a reference audio, the scoring results of the reference audio in multiple dimensions can be aggregated into one scoring result, that is, Figure 7 In the application, after the user scores a reference audio in four dimensions, namely, timbre, voice rhythm, pronunciation and articulation, the terminal device displays the summarized scoring result, which is a numerical value. The scoring results in the four dimensions are summarized to obtain a total scoring result. This calculation process can be completed by the terminal device or by the server. In this application, the scoring criteria and scoring results can also be in other forms, which are not limited in this application.

[0090] S1015. The terminal device sends the scoring results of the multiple reference audios to the server. Correspondingly, the server receives the scoring results of the multiple reference audios sent by the terminal device.

[0091] S1016. The server trains an acoustic feedback unit according to the user's scoring results for the multiple reference audios to obtain a trained acoustic feedback unit.

[0092] The server trains the acoustic feedback unit according to multiple reference audios and the scoring results of each of the multiple reference audios. Specifically, the server can extract the features of each of the multiple reference audios, train the acoustic feedback unit according to the features of each reference audio and the corresponding scoring results, and finally obtain a trained acoustic feedback unit.

[0093] Optionally, the server also includes a scoring result corresponding to each audio in the audio library, and the scoring result of each audio in the audio library is obtained by many different users scoring each audio in the audio library. The server can first perform pre-training based on each audio in the audio library and at least one scoring result corresponding to each audio. On the basis of the pre-training, the server then performs training in combination with multiple reference audios and the user's scoring results for each reference audio in the multiple reference audios, and finally obtains a trained acoustic feedback unit.

[0094] It can be seen that the acoustic feedback unit is trained based on multiple reference audios and the user's scoring results for the multiple reference audios, wherein the multiple reference audios are obtained by screening based on the user's feedback information. Therefore, the trained acoustic feedback unit reflects and embodies the user's preference for different audios. Therefore, the trained acoustic feedback unit can be used as a model or standard for judging the user's preference for different audios, and is used to measure the user's auditory perception of different audios. For example, an audio is input into the trained acoustic feedback unit, and the trained acoustic feedback unit can extract the features of the audio, score the audio based on the extracted features of the audio, and obtain the scoring result of the audio, and the scoring result of the audio reflects the user's preference for the audio.

[0095] In another implementation, the user does not need to input feedback information, the server directly sends multiple reference audios to the terminal device, the user scores the multiple reference audios through the terminal device, the terminal device sends the scoring results to the server, and the server trains the acoustic feedback unit based on the user's scoring results for the multiple reference audios to obtain a trained acoustic feedback unit. The trained acoustic feedback unit reflects and embodies the user's preference for different audios, so the trained acoustic feedback unit can be used as a model or standard for judging the user's preference for different audios, and is used to measure the user's auditory perception of different audios.

[0096] S102: The server obtains a first voice data set input by a user through a terminal device.

[0097] The first voice data set includes one or more first voices of the user, and the first voice can be a voice recorded by the user through a terminal device, or a voice of the user obtained by other means. The user sends the first voice data set to the server through the terminal device, and correspondingly, the server receives the first voice data set sent by the terminal device.

[0098] S103: The server inputs the first voice data set of the user into the voice cloning model. The voice cloning model generates a third voice data set according to the first voice data set. The third voice data set includes one or more third voices. At least one of the timbre, voice rhythm, pronunciation mode, and articulation mode of the third voice is the same as the first voice of the user.

[0099] A voice cloning algorithm is provided in the voice cloning model, and the voice cloning model is used to clone the voice input into the voice cloning model through the voice cloning algorithm to obtain a voice similar to the input voice.

[0100] The server inputs the user's first voice data set into the voice cloning model, and the voice cloning model clones the first voice in the first voice data set to obtain a third voice data set, wherein at least one of the timbre, voice rhythm, pronunciation method, and articulation method of the third voice in the third voice data set is the same as the user's first voice. Among them, the voice rhythm includes the pitch, volume, rhythm, intonation, etc. in the voice.

[0101] S104: The server trains the voice cloning model based on the third voice data set and the trained acoustic feedback unit to obtain a trained voice cloning model.

[0102] The voice cloning model is used to clone the input voice to obtain the cloned voice, and the trained acoustic feedback unit is used to score the cloned voice. The process of training the voice cloning model is actually the process of reinforcement learning training of the acoustic feedback unit and the voice cloning model. Through reinforcement learning training of the acoustic feedback unit and the voice cloning model, the voice cloning model can output voices that meet user needs, thereby obtaining a trained voice cloning model.

[0103] The following introduces the training process of the voice cloning model.

[0104] The third speech data set is input into the trained acoustic feedback unit, the trained acoustic feedback unit scores the third speech included in the third speech data set, outputs the scoring result, and then inputs the scoring result into the speech cloning model, the speech cloning model optimizes the third speech according to the scoring result of the third speech, and obtains the optimized speech; the optimized speech is input into the trained acoustic feedback unit again, the trained acoustic feedback unit scores the optimized speech again, and obtains the scoring result of the optimized speech, and the speech cloning model generates a new round of optimized speech according to the scoring result of the optimized speech; ... until the generated optimized speech meets the needs of the user, for example, until the scoring result of the generated optimized speech exceeds the threshold score, then the training is stopped to obtain the trained speech cloning model. Among them, the threshold score can be freely set according to the actual situation, for example, the full score is 5 points, and the threshold score can be set to 4 points or 4.5 points.

[0105] Therefore, the training of the voice cloning model includes multiple rounds of iterations, see Figure 8 As shown in the schematic diagram, in each round of iterative training, the optimized speech generated by the speech cloning model in the previous round of iterative training is input into the trained acoustic feedback unit, the acoustic feedback unit scores the optimized speech generated by the speech cloning model in the previous round of iterative training, outputs the scoring result, the speech cloning model is optimized based on the scoring result, and the optimized speech of this round is obtained, and the optimized speech of this round (the output of the speech cloning model in this round) is used as the input of the acoustic feedback unit of the next round of iterative training. Among them, for the first round of iterative training, the input of the acoustic feedback unit is the third speech. In each round of iterative training, the optimized speech output by the speech cloning model may include one or more optimized speech. In the case where the optimized speech output by the speech cloning model includes multiple optimized speech, in the next round of iterative training, the acoustic feedback unit needs to score each of the multiple optimized speech, and then input the scoring result of each optimized speech into the speech cloning model. The speech cloning model performs optimization processing according to each optimized speech and the scoring result corresponding to each optimized speech, and generates a batch of new optimized speech; ... until the scoring result of the preset number of optimized speech in the batch of new optimized speech generated by the speech cloning model by the acoustic feedback unit exceeds the threshold score, the training is stopped, and the trained speech cloning model is obtained. Among them, the preset number can be one, or multiple or all of the batch of new optimized speech. The specific preset number can be set according to the actual application requirements, and this application does not limit it.

[0106] Optionally, in the speech cloning model, the scoring result output by the acoustic feedback unit is used as a parameter in the loss function of the speech cloning model, and the scoring result is used to influence the speech cloning model to generate optimized speech for the current round of iterative training.

[0107] It can be seen that in the present application, the voice cloning model is trained based on the user's first voice data set, the feedback information input by the user, and the user's scoring results for multiple reference audios, wherein the multiple reference audios are determined based on the feedback information input by the user. In other words, the voice cloning model takes into account the user's wishes during the training process. The user's wishes include the user's preference for different audios and auditory perception. At the same time, the usage scenarios of the voice, the language used by the voice, etc. are also taken into account. Therefore, the voice cloning model provided in the present application can better meet the needs of users, and the voice obtained using the voice cloning model can better meet the user's auditory perception.

[0108] It can be understood that the process of training the voice cloning model by the server is actually the process of the server establishing a mapping relationship between the text corresponding to the user's first voice data set and the voice features that meet the user's auditory perception / willingness based on the user's first voice data set. In other words, the trained voice cloning model saves the mapping relationship between the text corresponding to the user's first voice data set and the voice features that meet the user's auditory perception / willingness. Therefore, the trained voice cloning model can output a voice that meets the user's auditory perception / willingness based on the text input by the user and the saved mapping relationship.

[0109] Steps S101 to S104 introduce the training process of the voice cloning model. In one implementation, before training the voice cloning model, it is necessary to obtain the user's first voice data set and the text corresponding to the first voice data set. In the process of training the voice cloning model, a mapping relationship between the text corresponding to the first voice data set and the voice features that meet the user's auditory perception / intention is established based on the first voice data set. In another implementation, the voice cloning model has a voice recognition function and can recognize the text corresponding to the first voice data set based on the first voice data set. Therefore, before training the voice cloning model, it is only necessary to obtain the user's first voice data set. The voice cloning model determines the text corresponding to the first voice data set based on the first voice data set. In the process of training the voice cloning model, a mapping relationship between the text corresponding to the first voice data set and the voice features that meet the user's auditory perception / intention is established based on the first voice data set.

[0110] After obtaining the trained voice cloning model, the voice cloning model can be applied to any one or more of the audio book scenario, the virtual human field, and the video creation field. The voice cloning model includes a mapping relationship between the text corresponding to the first voice data set and the voice features that meet the user's auditory perception / willingness. The server can input the text information input by the user into the trained voice cloning model to generate a voice, and the text corresponding to the first voice is the text input by the user.

[0111] It can be seen that in the present application, in the training process of the voice cloning model, the needs and preferences of users are taken into consideration, and the feedback information input by the user and the scoring results of the user on multiple reference audios are added to the training process of the voice cloning model. The voice cloning model obtained through training can better meet the needs of users; the second voice obtained by processing the text information input by the user using the trained voice cloning model can better meet the user's auditory perception.

[0112] In addition, the traditional voice cloning model / voice cloning model is trained directly through cloning technology based on the recording input by the user. When the user does not like or is not satisfied with the voice generated by the trained voice cloning model / voice cloning model, the user can only re-record and retrain the voice cloning model / voice cloning model. This training method has poor controllability and does not have secondary processing and optimization capabilities. In the present application, when the user is dissatisfied or does not like the voice generated by the trained voice cloning model, the user does not need to re-record, and the voice cloning model can be optimized based on the user's feedback information and the user's scoring results for multiple reference audios. That is, the training method provided in the present application has secondary processing and optimization capabilities.

[0113] After obtaining the trained voice cloning model, the trained voice cloning model can be used in the speech synthesis service. Fig. 9 , Fig. 9 A flowchart of an application method of a voice cloning model provided in the present application, wherein the method includes but is not limited to the description of the following content.

[0114] S201. The user inputs target information via a terminal device.

[0115] The user inputs target information on the terminal device, and the terminal device receives the target information input by the user.

[0116] Among them, the target information can exist in the form of a document, for example, the target information can be a document in the form of txt, or a document in the form of doc or docx, or a document in the form of pdf, etc.; the target information can also exist in the form of a picture, for example, the text can be photographed by a terminal device, or the text can be collected by an image acquisition device, and the collected text can be input into the terminal device. The picture format can be jpeg format, png format, tif format, bmp format, etc.; the target information can also exist in other forms, for example, a slide show, a combination of text and image, a combination of image and slide, a combination of text and slide, etc. This application does not limit the form of existence of the target information.

[0117] S202: The terminal device sends the target information to the server, and correspondingly, the server receives the target information input by the user.

[0118] S203: The server inputs the target information into the trained voice cloning model to generate a second voice.

[0119] The voice cloning model generates a second voice according to the text in the target information and the mapping relationship between the text corresponding to the first voice data set and the voice features that meet the user's auditory perception / willingness.

[0120] The server inputs the target information into the trained voice cloning model, and the server converts the target information into voice based on the trained voice cloning model to generate a second voice, which is biased towards the voice that the user prefers. The training process of the voice cloning model is described in the above method embodiment, and will not be described here for the sake of simplicity of the specification.

[0121] S204: The server sends the second voice to the terminal device, and correspondingly, the terminal device receives the second voice sent by the server.

[0122] S205: The terminal device plays the second voice.

[0123] The terminal device receives the second voice sent by the server, and the terminal device can be used to play the second voice.

[0124] It can be seen that the second voice obtained by processing the text information input by the user using the trained voice cloning model is more in line with the user's auditory perception.

[0125] The present application provides a device 800 for generating a voice cloning model, see Fig.10 , Fig.10 This is a schematic diagram of the structure of a device 800 for generating a voice cloning model provided by the present application. The device 800 for generating a voice cloning model can be configured as a server in a method embodiment. The device 800 includes:

[0126] An acquisition module 810 is used to acquire the scoring results of the multiple reference audios input by the user through the terminal device;

[0127] The acoustic feedback module 820 is used to perform training based on multiple reference audios and user scoring results of the multiple reference audios to obtain an acoustic feedback unit, and the acoustic feedback unit is used to measure the user's auditory perception of different audios;

[0128] An acquisition module 810 is used to acquire a first voice data set input by a user through a terminal device;

[0129] The speech cloning module 830 is used to train the speech cloning model based on the first speech data set and the acoustic feedback unit to obtain a trained speech cloning model.

[0130] In a possible implementation, the acquisition module 810 is further used to obtain feedback information input by the user through the terminal device, where the feedback information includes one or more of the application scenario of the voice cloning model, the emotion category used by the voice cloning model, and the language generated by the voice cloning model; the screening module 840 is used to filter and obtain multiple reference audios from the audio library according to the feedback information; and the sending module 850 is used to send the multiple reference audios to the terminal device.

[0131] In a possible implementation, the speech cloning model is trained based on the first speech data set and the acoustic feedback unit, including multiple rounds of iterative training. In the current round of iterative training: the speech cloning module 830 is used to generate optimized speech; the acoustic feedback module 820 is used to input the optimized speech into the acoustic feedback unit, the acoustic feedback unit scores the optimized speech, and obtains the scoring result of the optimized speech. The scoring result of the optimized speech is used as the input of the speech cloning model in the next round of iterative training, and is used to influence the speech cloning model to generate the next round of optimized speech; wherein, in the first round of iterative training, the optimized speech is generated by the speech cloning model according to the first speech data set.

[0132] In a possible implementation, in the current round of iterative training, the scoring result of the optimized speech is used as a parameter in the loss function of the speech cloning model in the next round of iterative training to influence the loss function in the speech cloning model.

[0133] In possible implementations, the voice cloning model is applied to any one or more of the audiobook scenario, the virtual human field, and the video creation field.

[0134] In a possible implementation, the acquisition module 810 is further used to receive target information input by a user, where the target information includes text; the voice cloning module 830 is further used to input the target information into a trained voice cloning model to generate a second voice.

[0135] In a possible implementation, the target information exists in the form of any one or more combinations of documents, pictures, and slides.

[0136] In a possible implementation, the scoring results of the multiple reference audios include the scoring results of each reference audio in the multiple reference audios in multiple dimensions, and the multiple dimensions include two or more dimensions of timbre, speech rhythm, pronunciation method and articulation method.

[0137] Among them, the acquisition module 810, the acoustic feedback module 820, the voice cloning module 830, the screening module 840 and the sending module 850 can all be implemented by software, or can be implemented by hardware. Exemplarily, the voice cloning module 830 is taken as an example to introduce the implementation of the voice cloning module 830. Similarly, the implementation of the acquisition module 810, the acoustic feedback module 820, the screening module 840 and the sending module 850 can refer to the implementation of the voice cloning module 830.

[0138] As an example of a software functional unit, the voice cloning module 830 may include code running on a computing device. The computing device may be a computing device in a cloud service, wherein the computing device may be, for example, a server, a virtual machine, a container, etc. Further, the computing device may be one or more. For example, the voice cloning module 830 may include code running on multiple computing devices. It should be noted that the multiple computing devices used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple computing devices used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Generally, a region may include multiple availability zones AZ.

[0139] Similarly, multiple computing devices used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC to achieve interconnection between VPCs through the communication gateway.

[0140] As an example of a hardware functional unit, the voice cloning module 830 may include at least one computing device. Alternatively, the voice cloning module 830 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0141] The multiple computing devices included in the voice cloning module 830 can be distributed in the same region or in different regions. The multiple computing devices included in the voice cloning module 830 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the voice cloning module 830 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0142] It should be noted that, in other embodiments, the voice cloning module 830 can be used to execute any step in a method for generating a voice cloning model, the acquisition module 810, the acoustic feedback module 820, the screening module 840 and the sending module 850 can all be used to execute any step in a method for generating a voice cloning model, and the steps that the acquisition module 810, the acoustic feedback module 820, the voice cloning module 830, the screening module 840 and the sending module 850 are responsible for implementing can be specified as needed. The acquisition module 810, the acoustic feedback module 820, the voice cloning module 830, the screening module 840 and the sending module 850 respectively implement different steps in a method for generating a voice cloning model to achieve all functions of the device 800 for generating a voice cloning model.

[0143] See also Fig.11 , Fig.11 This is a schematic diagram of the structure of a computing device provided in the present application. The computing device 900 may be, for example, a server, a virtual machine, a container, etc. The computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other through the bus 902. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0144] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 The bus 902 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 906, the processor 904, and the communication interface 908).

[0145] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0146] The memory 906 may include a volatile memory, such as a random access memory (RAM). The processor 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0147] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 810, acoustic feedback module 820, voice cloning module 830, screening module 840, and sending module 850, thereby implementing a method for generating a voice cloning model. That is, the memory 906 stores instructions for executing a method for generating a voice cloning model.

[0148] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network. Optionally, the acquisition module 810 and the sending module 850 may be located in the communication interface 908 .

[0149] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, a virtual machine, or a container, such as a central server, an edge server, or a sidecar container.

[0150] like Fig.12 As shown, Fig.12 A schematic diagram of the structure of a computing device cluster provided in the present application, wherein the computing device cluster includes at least one computing device 900, and the memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing a method for generating a voice cloning model.

[0151] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also respectively store partial instructions for executing a method for generating a voice cloning model. In other words, the combination of one or more computing devices 900 can be used to jointly execute instructions for a method for generating a voice cloning model.

[0152] When at least one computing device in the computing device cluster is configured as the apparatus 800 for generating a voice cloning model, the memory 906 in different computing devices 900 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the apparatus 800. That is, the instructions stored in the memory 906 in different computing devices 900 may implement the functions of one or more of the acquisition module 810, the acoustic feedback module 820, the voice cloning module 830, the screening module 840 and the sending module 850.

[0153] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.13 A schematic diagram of the structure of another computing device cluster is shown, such as Fig.13 As shown, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, the memory 906 in the computing device 900A stores instructions for the functions of the acquisition module 810 and the sending module 850. The computing device 900A is used to receive information sent by the user through the terminal device, such as the user's first voice data set, target information, feedback information input by the user, and the user's scoring results for multiple reference audios. The computing device 900A is also used to send information to the terminal device, such as sending multiple reference audios to the terminal device; the memory 906 in the computing device 900B stores instructions for executing the functions of the acoustic feedback module 820, the voice cloning module 830, and the screening module 840. The computing device 900B is used to process the information obtained in 900A, for example, according to the feedback information input by the user and the user's scoring results for multiple reference audios, the voice cloning model is trained to obtain the trained voice cloning model, and for another example, according to the target information input by the user and the trained voice cloning model, a second voice is generated.

[0154] It should be understood that Fig.13The functions of the computing device 900A shown in the figure may also be completed by multiple computing devices 900, or the computing device cluster may include multiple computing devices having the same functions as the computing device 900A. Similarly, the functions of the computing device 900B may also be completed by multiple computing devices 900, or the computing device cluster may include multiple computing devices having the same functions as the computing device 900B.

[0155] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.12 and Fig.13 The connection mode of the computing device cluster. Differently, the memory 906 in one or more computing devices 900 in the computing device cluster may store different instructions for executing a method for generating a voice cloning model. In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing a method for generating a voice cloning model. In other words, the combination of one or more computing devices 900 can jointly execute instructions for executing a method for generating a voice cloning model.

[0156] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform a method for generating a voice cloning model.

[0157] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device or a computing device cluster to execute a method for generating a voice cloning model.

[0158] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a voice cloning model, characterized in that: include: Obtaining scoring results of multiple reference audios input by a user through a terminal device; Performing training based on the multiple reference audios and the scoring results of the multiple reference audios by the user to obtain an acoustic feedback unit, wherein the acoustic feedback unit is used to measure the user's auditory perception of different audios; Acquire a first voice data set input by the user through the terminal device; Based on the first speech data set and the acoustic feedback unit, a speech cloning model is trained to obtain a trained speech cloning model.

2. The method according to claim 1, characterized in that Before obtaining the scoring results of the plurality of reference audios input by the user through the terminal device, the method further includes: Acquiring feedback information input by the user through the terminal device, the feedback information including one or more of an application scenario of the voice cloning model, an emotion category used by the voice cloning model, and a language generated by the voice cloning model; Filtering and obtaining the plurality of reference audios from an audio library according to the feedback information; The plurality of reference audios are sent to the terminal device.

3. The method according to claim 1 or 2, characterized in that: The voice cloning model is trained based on the first voice data set and the acoustic feedback unit, including multiple rounds of iterative training. In the current round of iterative training: The speech cloning model generates optimized speech; The optimized speech is input into the acoustic feedback unit, and the acoustic feedback unit scores the optimized speech to obtain a scoring result of the optimized speech, and the scoring result of the optimized speech is used as an input of the speech cloning model in the next round of iterative training to influence the speech cloning model to generate the next round of optimized speech; Wherein, in the first round of iterative training, the optimized speech is generated by the speech cloning model according to the first speech data set.

4. The method according to claim 3, characterized in that In the current round of iterative training, the scoring result of the optimized speech is used as a parameter in the loss function of the speech cloning model in the next round of iterative training, so as to affect the loss function in the speech cloning model.

5. The method according to any one of claims 1 to 4, characterized in that: The voice cloning model is applied to any one or more of the audio book scenario, the virtual human field, and the video creation field.

6. The method according to any one of claims 1 to 5, characterized in that: After obtaining the trained voice cloning model, the method further includes: The server receives target information input by the user, wherein the target information includes text; The target information is input into the trained voice cloning model to generate a second voice.

7. The method according to claim 6, characterized in that The target information exists in the form of any one or more combinations of documents, pictures and slides.

8. The method according to any one of claims 1 to 7, characterized in that: The scoring results of the multiple reference audios include scoring results of each reference audio in the multiple reference audios in multiple dimensions, and the multiple dimensions include two or more dimensions of timbre, speech rhythm, pronunciation method and articulation method.

9. A device for generating a voice cloning model, characterized in that: include: An acquisition module, used to acquire the scoring results of multiple reference audios input by the user through the terminal device; an acoustic feedback module, configured to perform training based on the multiple reference audios and the scoring results of the multiple reference audios by the user to obtain an acoustic feedback unit, wherein the acoustic feedback unit is configured to measure the user's auditory perception of different audios; The acquisition module is used to acquire a first voice data set input by the user through the terminal device; The speech cloning module is used to train the speech cloning model based on the first speech data set and the acoustic feedback unit to obtain a trained speech cloning model.

10. The device according to claim 9, characterized in that The acquisition module is further used to acquire feedback information input by the user through the terminal device, wherein the feedback information includes one or more of an application scenario of the voice cloning model, an emotion category used by the voice cloning model, and a language generated by the voice cloning model; A screening module, used for screening and obtaining the plurality of reference audios from an audio library according to the feedback information; A sending module is used to send the multiple reference audios to the terminal device.

11. The device according to claim 9 or 10, characterized in that The voice cloning model is trained based on the first voice data set and the acoustic feedback unit, including multiple rounds of iterative training. In the current round of iterative training: The voice cloning module is used to generate optimized voice; The acoustic feedback module is used to input the optimized speech into the acoustic feedback unit, and the acoustic feedback unit scores the optimized speech to obtain a scoring result of the optimized speech. The scoring result of the optimized speech is used as an input of the speech cloning model in the next round of iterative training to influence the speech cloning model to generate the next round of optimized speech; Wherein, in the first round of iterative training, the optimized speech is generated by the speech cloning model according to the first speech data set.

12. The device according to claim 11, characterized in that In the current round of iterative training, the scoring result of the optimized speech is used as a parameter in the loss function of the speech cloning model in the next round of iterative training, so as to affect the loss function in the speech cloning model.

13. The device according to any one of claims 9 to 12, characterized in that The voice cloning model is applied to any one or more of the audio book scenario, the virtual human field, and the video creation field.

14. The device according to any one of claims 9 to 13, characterized in that The acquisition module is also used to receive target information input by the user, wherein the target information includes text; The voice cloning module is also used to input the target information into the trained voice cloning model to generate a second voice.

15. The device according to claim 14, characterized in that The target information exists in the form of any one or more combinations of documents, pictures and slides.

16. The device according to any one of claims 9 to 15, characterized in that The scoring results of the multiple reference audios include scoring results of each reference audio in the multiple reference audios in multiple dimensions, and the multiple dimensions include two or more dimensions of timbre, speech rhythm, pronunciation method and articulation method.

17. A computing device cluster, characterized in that: The method comprises at least one computing device, wherein the at least one computing device comprises a memory and a processor, wherein the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 8.

18. A computer storage medium, characterized in that: The method comprises program instructions, which, when executed on a computing device cluster, enable the computing device cluster to execute the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The method comprises program instructions, which, when executed on a computing device cluster, enable the computing device cluster to execute the method according to any one of claims 1 to 8.

Images