Model training method and apparatus, device, medium, and program product

Abstract

A model training method, device, equipment, medium and program product are provided. The method comprises: obtaining a plurality of training samples comprising a first training sample and at least one second training sample, the first training sample comprising a first audio and a first text, and the second training sample comprising at least one second audio, the first audio and the second audio corresponding to a same first sound object and respectively corresponding to mutually different first language and second language; determining a first voiceprint feature and a second voiceprint feature corresponding to the first audio and the second audio respectively based on the plurality of training samples; selecting a target voiceprint feature from the first voiceprint feature or the second voiceprint feature based on a predetermined sampling strategy; and training a model based on the first text and the target voiceprint feature, so that the model generates a third audio of the first sound object based on the first text and the target voiceprint feature, the third audio corresponding to the first language.

Description

Technical Field

[0001] Examples of this disclosure relate to the field of computers and to model training methods, apparatus, devices, media, and program products. Background Technology

[0002] With the development of speech synthesis and speech generation technologies, speech cloning technology, which generates speech of a target speaker based on reference audio, has been widely used. In cross-language speech generation scenarios, the voiceprint information of the speaker can be obtained based on reference audio in one language, and combined with text in another language to generate speech with the corresponding timbre characteristics of the speaker. Summary of the Invention

[0003] In a first aspect of this paper, a model training method is provided. The method includes: acquiring multiple training samples, the multiple training samples including a first training sample and at least one second training sample, the first training sample including a first audio and a first text, the at least one second training sample including at least one second audio, the first audio and at least one second audio corresponding to a first vocal object, the first audio corresponding to a first language, and the at least one second audio corresponding to at least one second language, the first language being different from the second language; determining a first voiceprint feature and at least one second voiceprint feature based on the multiple training samples, the first voiceprint feature corresponding to the first audio, and the at least one second voiceprint feature corresponding to at least one second audio; selecting a target voiceprint feature from the first voiceprint feature or the at least one second voiceprint feature based on a predetermined sampling strategy; and training a model based on the first text and the target voiceprint feature, the model being configured to generate a third audio of the first vocal object based on the first text and the target voiceprint feature, the third audio corresponding to the first language.

[0004] In a second aspect of this paper, an apparatus for model training is provided. The apparatus includes: an acquisition module configured to acquire a plurality of training samples, the plurality of training samples including a first training sample and at least one second training sample, the first training sample including a first audio and a first text, the at least one second training sample including at least one second audio, the first audio and at least one second audio corresponding to a first vocal object, the first audio corresponding to a first language, the at least one second audio corresponding to at least one second language, and the first language being different from at least one second language; a feature determination module configured to determine a first voiceprint feature and at least one second voiceprint feature based on the plurality of training samples, the first voiceprint feature corresponding to the first audio, and the at least one second voiceprint feature corresponding to at least one second audio; a feature selection module configured to select a target voiceprint feature from the first voiceprint feature or the at least one second voiceprint feature based on a predetermined sampling strategy; and a model training module configured to train a model based on a first text and the target voiceprint feature, the model being configured to generate a third audio of the first vocal object based on the first text and the target voiceprint feature, the third audio corresponding to the first language.

[0005] In a third aspect of this document, an electronic device is provided. The electronic device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method of the first aspect when executed by the at least one processor.

[0006] In a fourth aspect of this document, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this document, a computer program product is provided. This computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] In this way, the model can use the voiceprint features of the same speaker for different languages ​​during training, reducing the influence of language attributes in the voiceprint features on the generation of target language audio, and maintaining the conditional control of voiceprint features on the timbre of the speaker.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the examples in this disclosure, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various examples in this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment based on some scenarios is shown; Figure 2 The diagram illustrates example processes for model training in several scenarios. Figure 3 A flowchart illustrating an example process for model training under several scenarios is shown; Figure 4 A schematic block diagram of an example device for model training under several scenarios is shown; and Figure 5 A block diagram is shown in which one or more examples of an electronic device can be implemented. Detailed Implementation

[0011] Examples of this disclosure will now be described in more detail with reference to the accompanying drawings. While some examples of this disclosure are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples set forth herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided in this disclosure are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of examples in this disclosure, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] The term "model" as used in this paper refers to a parameterized processing structure that can perform mapping, prediction, or generation processing based on input data and parameters, and may include structures based on attention mechanisms, state-space mechanisms, loop mechanisms, or other sequential processing mechanisms.

[0017] The term “training sample” as used in this paper refers to a data unit used to adjust model parameters, which may include one or more of the following: audio, text corresponding to the audio, language information, voiceprint features, speech rate features, and discrete speech representations.

[0018] As mentioned above, speech generation models can generate speech based on input text and voiceprint features corresponding to the speaker. For example, voiceprint features can be extracted from reference audio, and the text and voiceprint features can be used as conditions to input into the speech generation model to generate speech with the timbre features of the speaker corresponding to the reference audio. For speech generation models that use discrete speech representation, continuous speech can be converted into discrete speech representation, and the model can predict the corresponding discrete speech representation based on text and voiceprint conditions, and then the predicted discrete speech representation can be converted into audio.

[0019] In some cross-lingual speech generation scenarios, the voiceprint features extracted from reference audio may contain information related to the timbre of the speaker, as well as information related to the language, accent, or prosody of the reference audio. When the language of the reference audio differs from the language of the speech to be generated, the language-related information in the voiceprint features may affect the generation of the target language speech, making it difficult for the model to obtain speech that conforms to the characteristics of the target language while maintaining the timbre of the speaker.

[0020] Furthermore, the way a model utilizes different attributes in voiceprint features is usually related to the training data and training method. When the training data is limited or there is a strong correspondence between language and voiceprint features in the training samples, the model may have difficulty fully distinguishing between information related to the identity of the speaker and information related to language in the voiceprint features.

[0021] Therefore, how to make the model more effectively utilize the information related to the speaker in the voiceprint features during cross-language speech generation, and reduce the impact of the information related to the reference language on the target language generation results, is an issue that needs further consideration.

[0022] In view of this, a model training scheme is proposed. According to this scheme, the model training system acquires multiple training samples of the same first speaker in different languages ​​and determines the corresponding first voiceprint feature and at least one second voiceprint feature. The model training system selects target voiceprint features according to a predetermined sampling strategy, and then trains the model based on the first text and the target voiceprint features, so that the model generates a third audio in the first language corresponding to the first speaker.

[0023] The above approach allows the model to receive voiceprint features corresponding to different languages ​​during training while maintaining the correspondence between the speaker and the target language, and still generate audio in the first language. This reduces the model's reliance on language-related attributes in the voiceprint features, enabling it to utilize more information related to the speaker's identity and timbre. This weakens the transfer of reference language attributes to the target language generation during cross-language speech generation and improves the adaptability of cross-language speech generation. This process does not require adding a separate model branch for accent decoupling, thus avoiding increased model parameters and inference computation overhead.

[0024] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an example environment 100 based on some scenarios is shown. Figure 1 The training phase 130 and application phase 140 of model 150 are shown. A pre-training phase may be included before training phase 130, and a testing phase may be included after training phase 130, which is not shown in the figure.

[0026] Example environment 100 may include model training system 110 and model application system 120. During training phase 130, model training system 110 may train model 150 using training data 115. Training data 115 may include multiple training samples. Training samples may include one or more of the following: audio, corresponding text, language information, voiceprint features, speech rate features, and discrete speech representations. During training, input data and training objectives for model 150 may be determined based on the training samples, and the parameters of model 150 may be updated based on the difference between the prediction results of model 150 and the training objectives.

[0027] In some cases, Model 150 may be a parameterized processing structure configured to generate speech representations based on text and speaker features. Model 150 may include model structures based on attention mechanisms, state-space mechanisms, recurrent mechanisms, or other sequence processing mechanisms. In some examples, Model 150 may generate discrete speech representations based on input text, speaker features, and speech rate features, and the generated discrete speech representations may be further converted into audio.

[0028] In some cases, training data 115 may include training samples corresponding to multiple voice-producing objects. For the same voice-producing object, different training samples may include audio corresponding to different languages. Model training system 110 can determine the corresponding voiceprint features based on these training samples and select voiceprint features for training model 150 based on a predetermined sampling strategy. The organization of training samples, the determination of voiceprint features, and the training process of model 150 will be further explained later with reference to the accompanying drawings.

[0029] In some cases, a testing phase may be included after the training phase 130. The model training system 110 can use test data to test the trained model 150 to determine the performance of the model 150 in the speech generation task.

[0030] In the application phase 140, the trained model 150 can be provided to the model application system 120. The model application system 120 can acquire model input 125 and use model 150 to determine model output 126. For example, model input 125 may include target text and reference audio corresponding to the target speaker or voiceprint features determined based on the reference audio, and model output 126 may include target audio corresponding to the target speaker and the target language.

[0031] It should be understood that Figure 1The components and arrangement in the example environment 100 shown are for illustrative purposes only. The model training system 110 and the model application system 120 can be independent of each other or integrated into the same system or device. The model training system 110 and the model application system 120 can be deployed on one or more computing devices in a terminal device, server device, server cluster, distributed computing system, or cloud computing system.

[0032] In some cases, the model training system 110 and the model application system 120 can be implemented by a single computing device, or by multiple computing devices working together. For example, different computing devices can each handle training data processing, voiceprint feature extraction, model training, model inference, speech representation generation, or audio conversion. These implementation methods do not constitute a limitation on the scope of the solution.

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0034] The following description, in conjunction with the accompanying drawings, provides further examples. In the following text, the model training process is described primarily with respect to model training system 110, and the model application process is described with respect to model application system 120. The corresponding processing may also be performed collaboratively by one or more computing devices.

[0035] Figure 2 A schematic diagram of an example process 200 for model training is shown. Process 200 can be implemented in model training system 110.

[0036] In some scenarios, process 200 can be applied to cross-language speech generation or cross-language speech cloning. For example, voiceprint information can be obtained based on reference audio in one language corresponding to the speaker, and combined with text in another language to generate speech with the timbre characteristics of that speaker. In such scenarios, voiceprint features may carry information related to the speaker's identity and timbre, as well as information related to the language, accent, or prosody of the corresponding audio. To enable the model to make greater use of stable voiceprint information related to the speaker during cross-language generation, voiceprint features from different languages ​​corresponding to the same speaker can be included in the model training process.

[0037] In this paper, the speech object can be the object that generates the speech content in the audio. Different audio recordings can correspond to the same speech object or different speech objects. Through the speech object, object correspondences can be established between multiple audio recordings, the text corresponding to the multiple audio recordings, and the voiceprint features determined based on the multiple audio recordings. For example, when multiple training samples correspond to the same speech object, it can be determined that the audio recordings and corresponding voiceprint features in these training samples are all associated with that speech object.

[0038] In this paper, text can be a textual representation corresponding to the speech content in the audio. In some cases, text can be used to indicate the speech content to be generated by the model and the corresponding language. For example, the first text can correspond to the speech content in the first audio and can correspond to the first language. During model training, the first text can be used as one of the input conditions of the model to enable the model to generate a speech representation or audio corresponding to the first text.

[0039] like Figure 2 As shown in box 210, the model training system 110 can acquire multiple training samples. The multiple training samples may include a first training sample and at least one second training sample. The first training sample may include a first audio file and a first text file corresponding to the first audio file. The at least one second training sample may include at least one second audio file. The first audio file and the at least one second audio file may correspond to a first vocal object. The first audio file may correspond to a first language, and the at least one second audio file may each correspond to at least one second language, where each of the at least one second language may be different from the first language.

[0040] In this paper, training samples can be data units used to adjust model parameters. Training samples can include one or more of the following: audio, corresponding text, language information, voiceprint features, speech rate features, and discrete speech representations. Compared to training based solely on the voiceprint features of the current training audio, the aforementioned multiple training samples can provide audio and voiceprint information corresponding to different languages ​​for the same speaker, thus providing a data foundation for subsequent sampling of voiceprint features corresponding to different languages.

[0041] Voiceprint features are features obtained from audio to represent the voice of an individual. Voiceprint features can be represented as vectors and used as conditional inputs to a model. They can carry information related to the timbre of the voice, as well as information related to prosody, accent, language, or other pronunciation attributes within the corresponding audio. Therefore, even if multiple audio recordings correspond to the same voice, the voiceprint features corresponding to different audio recordings may differ due to variations in the language, prosody, or pronunciation style associated with each recording.

[0042] In some examples, the first voiceprint feature, the second voiceprint feature, and the third voiceprint feature described later may include voiceprint features determined based on a single sentence audio, which may also be called sentence-by-sentence voiceprint features. Sentence-by-sentence voiceprint features can be used to characterize the speaker corresponding to the audio and can preserve fine-grained speech attributes in that audio.

[0043] In some cases, language can be used to represent the language to which audio or text corresponds. For the same speaker, different training samples can correspond to different languages. For example, the first audio can correspond to the first language, and at least one second audio can correspond to at least one second language different from the first language. Thus, while keeping the speaker consistent, the model can be provided with voiceprint features corresponding to different languages.

[0044] In some cases, the first audio and at least one second audio can come from existing training data. In other cases, if real audio of the first speaker for one or more second languages ​​is lacking, at least one second audio can be generated to construct training samples of the same speaker for different languages.

[0045] Specifically, the model training system 110 can acquire at least one second text. The at least one second text corresponds to at least one second language. For example, the second text can be text determined based on the second language to be generated, or it can be obtained by converting existing text into the corresponding second language. In some examples, the second text can also be associated with language information used to indicate the corresponding second language.

[0046] The model training system 110 can generate at least one second audio file using a first model, based on global voiceprint features corresponding to the first voice-speaker and at least one second text. The global voiceprint features can be determined based on multiple voiceprint features corresponding to the first voice-speaker. Therefore, a second audio file corresponding to both the first voice-speaker and the second language can be obtained without the first voice-speaker actually recording the corresponding second language audio, thus providing corresponding training data for subsequent cross-language voiceprint sampling training.

[0047] In some cases, global voiceprint features can be object-level voiceprint representations determined based on multiple voiceprint features of the same speaker. For example, voiceprint features can be extracted separately from multiple audio recordings corresponding to a first speaker, these features can be aggregated, and the aggregation result can be normalized to obtain global voiceprint features. Compared with voiceprint features determined based on a single audio recording, global voiceprint features can better represent voiceprint information that remains relatively stable across multiple audio recordings and reduce the influence of language, accent, or prosody variations corresponding to specific sentences on voiceprint representation.

[0048] In some cases, the first model can generate a discrete speech representation of the corresponding second language based on the second text and global speaker features. Specifically, the model training system 110 can input at least one second text and global speaker features into the first model to obtain at least one discrete speech representation corresponding to at least one second language, and further convert the at least one discrete speech representation into at least one second audio.

[0049] In this paper, discrete speech representation can be a sequence of representations obtained by discretizing a continuous speech signal, or it can be called speech tokens. For example, a speech coding component can be used to map a continuous speech signal into one or more discrete representations, and a vocoder or other audio generation component can be used to convert the discrete speech representation into a time-domain audio waveform.

[0050] In some cases, the first model used to generate the second audio can be obtained by further training the base model. The base model can be a model that already has speech generation capabilities before the aforementioned further training. For example, the base model can be configured to generate corresponding speech representations based on text and voiceprint features. The first model can be a model obtained by further training the base model based on global voiceprint features, and can be used to generate corresponding audio based on text and global voiceprint features.

[0051] In some cases, the model training system 110 can acquire multiple third training samples. These multiple third training samples may include multiple fourth audio recordings and multiple third texts corresponding to each of the fourth audio recordings. The multiple fourth audio recordings may correspond to a first speaker. For example, the multiple fourth audio recordings may be multiple audio recordings of the first speaker expressing different speech content. The model training system 110 can determine the global voiceprint features corresponding to the first speaker based on the multiple fourth audio recordings, and train a base model based on the multiple third texts and the global voiceprint features to obtain a first model.

[0052] In some cases, to determine global voiceprint features, the model training system 110 can extract multiple third voiceprint features from multiple fourth audio sources. Each third voiceprint feature can be a sentence-by-sentence voiceprint feature extracted from the corresponding fourth audio source. Sentence-by-sentence voiceprint features can be represented as embedding vectors, for example, denoted as utt_embedding. In some examples, the third voiceprint feature can have 192 dimensions. It should be understood that 192 dimensions is merely an example, and the third voiceprint feature can have other appropriate dimensions.

[0053] In some examples, the model training system 110 can aggregate multiple third voiceprint features to obtain aggregated voiceprint features, and then normalize these aggregated voiceprint features to obtain global voiceprint features. In this paper, aggregated voiceprint features can be intermediate voiceprint representations obtained by aggregating multiple voiceprint features corresponding to the same voice-producing object, but which have not yet undergone normalization. For example, for N third voiceprint features corresponding to the first voice-producing object s, the model training system 110 can determine the aggregated voiceprint features using the following formula:

[0054] Where spk_emb_s represents the aggregated voiceprint feature corresponding to the first voice object s, utt_emb_i represents the third voiceprint feature corresponding to the i-th audio, and N represents the number of third voiceprint features participating in the aggregation.

[0055] In some examples, the model training system 110 can further normalize the aggregated voiceprint features to obtain the global voiceprint features corresponding to the first voice-emitting object. For example, the global voiceprint features can be determined by the following formula:

[0056] in, The L2 norm represents the aggregated voiceprint features. The global voiceprint features can also be denoted as spk_embedding. Normalization can make the global voiceprint features have a more consistent vector scale, so as to serve as the voiceprint condition of the model.

[0057] In some cases, the method of aggregating multiple third voiceprint features is not limited to arithmetic averaging. For example, median, trimmed average, weighted average, or cluster centers can also be used to obtain aggregated voiceprint features. When using a weighted average, the corresponding weights can be determined based on the audio quality, signal-to-noise ratio, or other quality information corresponding to each of the multiple fourth audio frequencies.

[0058] By aggregating multiple third voiceprint features corresponding to the same first voice source, the influence of specific language, accent, rhythm, or local expressive information in different fourth audio frequencies on the object-level voiceprint representation can be reduced, allowing the resulting global voiceprint features to reflect more of the relatively stable voiceprint information among multiple fourth audio frequencies. Further normalization processing can give different voiceprint features a more consistent vector scale, thus facilitating the use of global voiceprint features as conditional input to the model.

[0059] After obtaining the global voiceprint features, the model training system 110 can use these global voiceprint features to further train the base model. Specifically, for each of the multiple third training samples, the model training system 110 can replace the third voiceprint features corresponding to the third training sample with global voiceprint features, and train the base model based on the multiple third texts and the corresponding global voiceprint features to obtain the first model.

[0060] For example, for a third training sample consisting of third text and a corresponding fourth audio, the third voiceprint feature extracted from the corresponding fourth audio can be used as a voiceprint condition during the existing training of the base model. During further training of the base model, this third voiceprint feature can be discontinued as the voiceprint condition; instead, the global voiceprint feature corresponding to the first speaker can be used. Therefore, for multiple third training samples corresponding to the first speaker, even if the text content, speech content, or language differs in different third training samples, the same global voiceprint feature can be used as the voiceprint condition provided to the base model.

[0061] In some cases, the further training described above can utilize the existing model structure and training framework of the base model. For example, the network structure of the base model can remain unchanged, while adjusting the source of the voiceprint conditions provided to the base model during training. During training, a warmup-stable-decay (WSD) learning rate scheduling strategy can be used, or other appropriate learning rate scheduling methods can be employed.

[0062] In some cases, embedding vector discarding can also be performed on one or more conditional features during the training of the first model. For example, the model training system 110 can replace the voiceprint conditional feature with a zero vector of the same dimension as the voiceprint conditional feature according to a predetermined probability, and can also replace the speech rate feature with a zero vector of the same dimension as the speech rate feature according to another predetermined probability.

[0063] In some cases, after obtaining the first model, the model training system 110 can use the first model to generate at least one second audio. Specifically, the model training system 110 can acquire at least one second text, which corresponds to at least one second language. In some examples, the second text can be obtained by converting existing text into the second language. The second text can also be associated with language information indicating the corresponding second language.

[0064] The model training system 110 can generate at least one second audio file based on at least one second text and global speaker features corresponding to the first speaker, using a first model. For example, at least one second text and global speaker features can be input into the first model to obtain at least one discrete speech representation corresponding to at least one second language, and the at least one discrete speech representation can be converted into at least one second audio file.

[0065] In this paper, discrete speech representation can be a sequence of representations obtained by discretizing a continuous speech signal, also known as a speech token. For example, discrete speech representation can be obtained by mapping continuous speech representations to discrete codewords in a predetermined codebook. A first model can predict one or more discrete speech representations based on a second text and global speaker features, and a vocoder can convert the predicted discrete speech representations into a time-domain audio waveform to obtain the second audio. In this paper, the vocoder can be a model or processing component configured to convert discrete speech representations or other acoustic representations into audio waveforms.

[0066] For example, if the first speaker already has audio corresponding to the first language but lacks audio corresponding to the second language, a second text corresponding to the second language can be obtained, and the second text and the global voiceprint features corresponding to the first speaker can be provided to the first model. The first model can generate a discrete speech representation corresponding to the second text and the second language, and further obtain a second audio based on this discrete speech representation. Thus, the second audio can correspond to the first speaker and the second language without the first speaker actually recording audio in the second language.

[0067] Using the above method, a second audio corresponding to one or more second languages ​​can be generated for a first voice subject, thereby constructing training data corresponding to different languages ​​for the same first voice subject. Since global voiceprint features determined based on multiple voiceprint features are used as object-level voiceprint conditions when generating the second audio, the influence of language or accent attributes in a single source audio on the generated second audio can be reduced, thus making the generated second audio suitable for further use in constructing training samples required for cross-language voiceprint sampling training.

[0068] After generating at least one second audio recording, the model training system 110 can extract at least one second voiceprint feature from each of the at least one second audio recording. The second voiceprint feature can be a sentence-by-sentence voiceprint feature corresponding to the respective second audio recording, and can correspond to a specific second language. Thus, for the same first speaker, a first voiceprint feature corresponding to the first language and one or more second voiceprint features corresponding to the second language can be obtained.

[0069] In some cases, the generated second audio, along with the corresponding second text, discrete speech representation, and second voiceprint features, can form pseudo-parallel data together with existing training data. In this paper, pseudo-parallel data can be audio and related data of the same speaker in different languages ​​obtained through speech generation processing, and can be used to establish the correspondence between training samples of the same speaker in multiple languages.

[0070] For example, at least one second training sample corresponding to at least one vocal object can be represented by the following formula:

[0071] Where s represents the vocal object, such as the first vocal object, S represents the set of vocal objects, which may include one or more vocal objects, and second training samples corresponding to one or more languages ​​can be constructed for different vocal objects. l represents the language representation language corresponding to the corresponding training sample. Represents a set of languages. This represents the text corresponding to the audio. This represents the discrete speech representation corresponding to the given audio. This represents the sentence-by-sentence voiceprint features extracted from the corresponding audio.

[0072] It should be understood that the second audio in at least one second training sample does not require actual recording by the corresponding speaker. By further determining the second voiceprint features from the second audio, subsequent training can obtain multiple sentence-by-sentence voiceprint features corresponding to the same speaker in different languages, thereby providing a data basis for selecting among the voiceprint features corresponding to different languages ​​according to a predetermined sampling strategy.

[0073] Alternatively, at least one second audio file can also be obtained through other speech generation or speech conversion methods. For example, the audio corresponding to the second language can be converted into a second audio file with the voiceprint attributes of the first speaker through speech conversion (VC); other text-to-speech models can also be used to generate the second audio file; or, multiple speech generation methods can be combined to obtain second audio files corresponding to different second languages.

[0074] In some cases, second audio corresponding to multiple second languages ​​can be generated in parallel. For example, global voiceprint features corresponding to the first speaker can be shared among multiple generation tasks, and corresponding second audio can be generated using second text corresponding to different second languages.

[0075] In some cases, the base model and the first model can adopt model structures based on attention mechanisms, state-space mechanisms, recurrent mechanisms, or other sequence processing mechanisms. The speech discretization component used to obtain the discrete representation of speech can adopt different codebook structures, and the feature extraction component used to determine speaker features can adopt convolutional, temporal, self-supervised, or other feature extraction structures. As long as the corresponding model can generate the corresponding speech representation or audio based on text and speaker conditions, the specific model structure, speech representation method, and feature extraction method are not restricted.

[0076] Continue to refer to Figure 2In some cases, the model training system 110 can determine the source of the target voiceprint feature based on a predetermined sampling strategy. In this paper, the predetermined sampling strategy can be a selection rule that is pre-set or adjusted during model training, used to determine the probability or probability distribution of different voiceprint features being selected as target voiceprint features. Through the predetermined sampling strategy, the relationship between the language corresponding to the target voiceprint feature and the first language corresponding to the first audio can be changed while maintaining the fact that all voiceprint features correspond to the first vocal object.

[0077] In some examples, the model training system 110 may perform the above sampling on a first training sample. The first training sample may correspond to a first vocal object, a first language, and a first text, and may have a first voiceprint feature extracted from the first audio. At least one second training sample may also correspond to the first vocal object, but to at least one second language different from the first language, and may each have at least one second voiceprint feature extracted from the corresponding second audio.

[0078] When constructing training data for training model 220, model training system 110 may, according to a first probability 211, select a second training sample from data in at least one second language that corresponds to the first speaker but is different from the first language, and use the second voiceprint feature corresponding to the second training sample as the target voiceprint feature. In other words, in this sampling branch, the voiceprint feature input to model 220 comes from the audio of the first speaker in another language, while the first text and the first audio used to supervise model training still correspond to the first language.

[0079] In some examples, if multiple second languages ​​exist, the model training system 110 can randomly select one second language from the multiple second language data corresponding to the first speaker, and further select a second voiceprint feature corresponding to that second language. For example, a second voiceprint feature can be uniformly and randomly selected from a set of second voiceprint features that satisfy the condition that the language is different from the first language.

[0080] For example, such as Figure 2 As shown, the predetermined sampling strategy can indicate a first probability 211 and a second probability 212. The first probability 211 can correspond to the probability of selecting a target voiceprint feature from at least one second voiceprint feature, and the second probability 212 can correspond to the probability of selecting a first voiceprint feature as the target voiceprint feature.

[0081] Accordingly, if the sampling result corresponds to the first probability 211, the model training system 110 can select a second voiceprint feature from the above set and provide the selected second voiceprint feature as the target voiceprint feature to the model 220. At this time, the voice-producing object corresponding to the target voiceprint feature is the same as the voice-producing object corresponding to the first audio, but the language corresponding to the target voiceprint feature is different from the first language corresponding to the first audio.

[0082] If the sampling result corresponds to the second probability 212, the model training system 110 can select the first voiceprint feature corresponding to the first audio as the target voiceprint feature and provide the first voiceprint feature to the model 220. At this time, both the target voiceprint feature and the first audio correspond to the first speaker and the first language, which is equivalent to maintaining the training method based on the voiceprint features of the current training samples.

[0083] In some examples, the above probability sampling relationship can be expressed by the following formula: (4) in, This represents the target voiceprint features of input model 220. This indicates a voiceprint feature selected from at least one second voiceprint feature. This represents the first voiceprint feature corresponding to the first audio audio. Indicates the first probability. This represents the second probability.

[0084] It should be understood that in the sampling branch corresponding to the first probability 211, what changes is the source of the voiceprint conditions provided to the model 220, while the first text corresponding to the first training sample and the training target remain unchanged. That is, even if the target voiceprint features come from the second language audio of the first speaker, the model 220 can still generate a speech representation corresponding to the first language based on the first text, and use the speech information corresponding to the first audio as the training target. Thus, while maintaining the consistency of the speaker's identity and the current training target, the language corresponding to the voiceprint conditions can be made inconsistent with the language corresponding to the current training sample.

[0085] By employing the sampling method described above, model 220 can avoid relying on a fixed correspondence where "the language corresponding to the voiceprint feature is consistent with the language to be generated" for training. For example, if the first training sample corresponds to the first language, but the target voiceprint feature comes from the second language, model 220 still needs to generate the speech of the first speaker in the first language. This reduces model 220's dependence on language-related attributes in the target voiceprint feature and encourages model 220 to utilize more shared information related to the identity and timbre of the first speaker in voiceprint features from different languages.

[0086] In some cases, the first probability falls within a predetermined probability range of 0.3 to 0.7, and the second probability decreases as the first probability increases. When the first probability is higher, the model 220 receives more training iterations of cross-lingual voiceprint features; when the first probability is lower, the model 220 receives more training iterations of current audio voiceprint features. By adjusting the first probability, the ratio of cross-lingual voiceprint conditions to current language voiceprint conditions used during training can be adjusted, thereby striking a balance between enhancing the model's adaptability to cross-lingual voiceprint conditions and preserving the speech attributes related to the speaker in each sentence's voiceprint.

[0087] In some cases, where at least one second language includes multiple second languages, the model training system 110 can determine the first probability corresponding to each second language based on the similarity between the first language and each of the multiple second languages, and select the target voiceprint feature from multiple second voiceprint features based on each first probability.

[0088] In this way, the probability of the corresponding second voiceprint features participating in training can be adjusted according to the relationship between different second languages ​​and the first language, so that the model 220 can obtain different degrees of cross-language voiceprint conditions under different language combinations, thereby improving the adaptability of the predetermined sampling strategy to multilingual training scenarios.

[0089] In some cases, the model training system 110 can also adjust the first probability based on the training progress, so that the first probability increases as the training progresses. For example, in the early stages of training, a relatively small first probability can be set so that more training samples use the first voiceprint feature corresponding to the current first audio as the target voiceprint feature; as the training progresses, the first probability can be gradually increased so that more training samples use the second voiceprint feature corresponding to the second language as the target voiceprint feature.

[0090] In this way, training methods based on the current audio voiceprint features can be retained more in the early stages of training, while the proportion of cross-lingual voiceprint conditions used can be gradually increased as training progresses, thereby gradually enhancing the adaptability of model 220 to cross-lingual voiceprint conditions. The model training system 110 trains model 220 based on the first text and target voiceprint features. Model 220 can also receive speech rate features determined based on the first audio and output a discrete representation of the predicted speech.

[0091] Continue to refer to Figure 2After determining the target voiceprint features, the model training system 110 can train the model 220 based on the target voiceprint features. In some cases, when performing the above-mentioned cross-lingual voiceprint sampling training, the model training system 110 can initialize the model 220 from the model parameters of the base model. This base model can be a model used before training the first model, rather than the first model itself. Since the first model has been further trained based on global voiceprint features, its utilization of some prosody or expressive information in sentence-by-sentence voiceprint features may be relatively weakened. Therefore, initializing the model 220 with the parameters of the base model can further reduce the model's dependence on language or accent attributes in voiceprint features while retaining the existing speech generation capabilities of the base model.

[0092] In some cases, the model training system 110 can also determine speech rate features based on the first audio. In this paper, speech rate features can be conditional features used to characterize the speech speed in the corresponding audio, and can be represented as an embedding vector, for example, denoted as spkrate_embedding. Speech rate features can be used together with the first text and the target voiceprint features as conditional inputs to model 220, so that model 220 references the speech rate information corresponding to the first audio when generating speech representations.

[0093] In some cases, the model training system 110 can input the first text, target voiceprint features, and speech rate features into the model 220 to obtain a predicted discrete speech representation. In box 230, the model 220 can generate a third audio corresponding to the first speaker based on the first text and target voiceprint features; the third audio corresponds to the first language. In some examples, the model 220 can first generate a predicted discrete speech representation corresponding to the third audio, and then further generate the third audio based on this predicted discrete speech representation.

[0094] In some cases, model 220 may not directly output the audio waveform, but instead first generate a discrete representation of the predicted speech, and then further generate a third audio based on the discrete representation of the predicted speech. For example, a vocoder can be used to convert the discrete representation of the predicted speech into a third audio. It should be understood that box 230 is used to illustrate that model 220 can generate a third audio corresponding to the first language, and does not require that the audio waveform be actually generated before each parameter update during model training.

[0095] In box 240, the model training system 110 can determine the discrete representation of the target speech based on the first audio, and update the parameters of model 220 based on the difference between the predicted discrete speech representation and the target discrete speech representation. Since the target discrete speech representation is determined based on the first audio, the prediction target of model 220 in the current training samples does not change due to changes in the source of the target voiceprint features. That is, when the target voiceprint features come from the second audio corresponding to the second language, model 220 still needs to predict the discrete speech representation of the first language based on the first text, and use the target discrete speech representation corresponding to the first audio as training supervision.

[0096] In this way, the language source of the voiceprint conditions can be changed without altering the training objective corresponding to the first text and the first audio, enabling model 220 to learn to generate speech in the current language under different language voiceprint conditions for the same speaker. This training process is mainly achieved by adjusting the combination of voiceprint conditions in the training data, without needing to set up an independent decoupling model or dedicated network branch for language attributes. Thus, cross-language voiceprint training can be achieved while maintaining the basic model structure.

[0097] In some cases, the model training system 110 can train the model 220 using the same or similar optimizer and training configuration as the base model. For example, the learning rate can employ a warm-up, stable decaying learning rate scheduling strategy. Embedding vector discarding can also be performed on the target voiceprint features and speech rate features during training. For example, the target voiceprint features can be replaced with a first zero vector of the same dimension with a first discard probability, and the speech rate features can be replaced with a second zero vector of the same dimension with a second discard probability. As a specific example, the first discard probability can be 10%, and the second discard probability can be 50%.

[0098] In some cases, the input sequence used to train model 220 may include a text portion corresponding to the first text and a speech portion corresponding to the first audio. Model training system 110 may determine the training loss based solely on the speech portion. For example, the label corresponding to the text portion may be set to a predetermined value, such as -100, that does not participate in the loss calculation, and the cross-entropy loss may be determined based on the predicted discrete speech representation and the target discrete speech representation at the corresponding position of the speech portion to update the parameters of model 220.

[0099] In the application phase 140, the model application system 120 can acquire the target text and reference audio. The reference audio corresponds to the target speaker, and the target text corresponds to the target language. The model application system 120 can determine the third and fourth voiceprint features corresponding to the target speaker based on the reference audio. The third voiceprint feature can be the sentence-by-sentence voiceprint feature corresponding to the reference audio, and the fourth voiceprint feature can be the global voiceprint feature corresponding to the target speaker.

[0100] The model application system 120 can determine voiceprint conditions based on at least one of the third and fourth voiceprint features, and generate target audio using a machine learning model 150 based on the target text and the voiceprint conditions. The target audio corresponds to the target speaker and the target language.

[0101] In some cases, the model application system 120 may choose a third voiceprint feature as a voiceprint condition to preserve more of the sentence-by-sentence timbre, rhythm or accent attributes in the reference audio; or it may choose a fourth voiceprint feature as a voiceprint condition to reduce the influence of the language or accent attributes of the reference audio on the target audio.

[0102] In some cases, the model application system 120 can also fuse the third and fourth voiceprint features based on the first control parameter to obtain fused voiceprint features, and use the fused voiceprint features as voiceprint conditions. The first control parameter can determine the fusion ratio of the third and fourth voiceprint features in the fused voiceprint features, thereby adjusting the degree of accent preservation in the target audio.

[0103] For example, the fused voiceprint features can be determined using the following formula:

[0104] Where, `emb_final` represents the fused voiceprint features before normalization. `utt_emb` represents the third voiceprint feature, and `spk_emb` represents the fourth voiceprint feature. `α` represents the first control parameter, which can be located in a closed interval between 0 and 1. At that time, more of the voiceprint features can be preserved; when When, global voiceprint features can be used; when When it is between 0 and 1, it can be continuously adjusted between the two.

[0105] In some examples, the fused voiceprint features can also be normalized to obtain voiceprint conditions for inputting machine learning model 150.

[0106] In this way, different degrees of accent preservation or reduction can be achieved by adjusting the voiceprint conditions without training multiple models separately for different degrees of accent preservation, while taking into account both the timbre preservation of the target speaker and the generation of the target language speech.

[0107] In one example training flow, the model training system 110 can sequentially perform training data preparation, global voiceprint feature determination, first model training, pseudo-parallel data generation, sentence-by-sentence voiceprint feature extraction, and... Figure 2The cross-language voiceprint sampling training shown is used to obtain a trained machine learning model 150. After training, the machine learning model 150 can be provided to the model application system 120 for speech generation processing in the application phase 140.

[0108] In one example inference process, the model application system 120 can acquire the target text and reference audio, and determine the sentence-by-sentence voiceprint features and global voiceprint features corresponding to the target speaker. The model application system 120 can determine voiceprint conditional features from the sentence-by-sentence voiceprint features, global voiceprint features, or a fusion of both, based on the accent control method. Subsequently, the model application system 120 can input the target text, voiceprint conditional features, and optional speech rate features into the machine learning model 150 to generate a discrete speech representation, and further utilize a vocoder to convert the discrete speech representation into the target audio.

[0109] Figure 3 A flowchart of an example process 300 for model training under certain conditions is shown. Process 300 can be implemented at model training system 110. See below for reference. Figure 3 To describe process 300.

[0110] like Figure 3 As shown in box 310, the model training system 110 acquires multiple training samples. These multiple training samples include a first training sample and at least one second training sample. The first training sample includes a first audio and a first text, and the at least one second training sample includes at least one second audio. The first audio and at least one second audio correspond to a first vocal object, the first audio corresponds to a first language, and the at least one second audio corresponds to at least one second language. The first language is different from the second language.

[0111] In box 320, the model training system 110 determines a first voiceprint feature and at least one second voiceprint feature based on multiple training samples. The first voiceprint feature corresponds to a first audio audio, and the at least one second voiceprint feature corresponds to at least one second audio audio.

[0112] In box 330, the model training system 110 selects a target voiceprint feature from a first voiceprint feature or at least one second voiceprint feature based on a predetermined sampling strategy.

[0113] In box 340, model training system 110 trains a model based on first text and target voiceprint features. The model is configured to generate a third audio of a first voice object based on the first text and target voiceprint features, the third audio corresponding to the first language.

[0114] In this way, the model training system 110 can train the model under cross-lingual voiceprint conditions of the same speaker and enable the model to generate a third audio in the first language based on the first text. Cross-lingual voiceprint sampling changes the combination of conditions during training, thereby reducing the model's dependence on the language attributes in the voiceprint. This process does not require adding a dedicated decoupling branch, thus reducing the overhead of additional parameter storage and inference computation.

[0115] In some cases, the predetermined sampling strategy indicates a first probability and a second probability, and selecting the target voiceprint feature includes selecting the target voiceprint feature from at least one second voiceprint feature based on the first probability. The model training system 110 can also select the first voiceprint feature as the target voiceprint feature based on the second probability. In this way, the model training system 110 can select between cross-lingual voiceprint features and current-language voiceprint features according to two types of probabilities. Both paths reuse the same training process, thereby controlling the allocation of training samples for both types and reducing the computational overhead of additional branches.

[0116] In some cases, the first probability can fall within a predetermined probability range, and the second probability can decrease as the first probability increases. In this way, the predetermined probability range can limit the cross-lingual sampling ratio and create a linkage between the two types of sampling probabilities. Controlled probability changes can reduce training path imbalance, thereby reducing parameter update fluctuations and improving training stability.

[0117] In some cases, at least one second language includes multiple second languages. At least one second voiceprint feature includes multiple second voiceprint features corresponding to the multiple second languages ​​respectively, and selecting a target voiceprint feature from at least one second voiceprint feature includes: determining a first probability corresponding to each second language based on the similarity between the first language and each of the multiple second languages. The model training system 110 can also select a target voiceprint feature from the multiple second voiceprint features based on the first probability corresponding to each second language. In this way, the model training system 110 can adjust the sampling opportunities for each second language according to language similarity. Assigning samples according to language relationships can reduce invalid or duplicate sampling, thereby improving the utilization rate of effective samples and training processing efficiency.

[0118] In some cases, the model training system 110 can adjust the first probability based on the training progress, so that the first probability increases as the training progresses. In this way, the model training system 110 can retain more current language conditions in the early stages of training and add cross-language conditions in the later stages of training. Gradually changing the training distribution can reduce large gradient changes in the early stages, thereby reducing the fluctuation of model parameter updates.

[0119] In some cases, at least one second audio is generated through the following steps: acquiring at least one second text, which corresponds to at least one second language. The model training system 110 can also generate at least one second audio using a first model based on global voiceprint features corresponding to the first speaker and at least one second text, whereby the global voiceprint features are determined based on multiple voiceprint features corresponding to the first speaker. In this way, the model training system 110 can generate second audio in different languages ​​using object-level global voiceprints, forming cross-language corresponding training data. Multiple generation tasks can reuse the first model and global voiceprint conditions, thereby reducing model reloading and improving data generation throughput.

[0120] In some cases, generating at least one second audio file using the first model includes: inputting at least one second text and global speaker features into the first model to obtain at least one discrete speech representation corresponding to at least one second language. The model training system 110 can also convert at least one discrete speech representation into at least one second audio file.

[0121] In some cases, the first model is trained through the following steps: acquiring multiple third training samples, which include multiple fourth audio recordings and multiple third text recordings, with the multiple fourth audio recordings corresponding to the first speaker. The model training system 110 can also determine global voiceprint features based on the multiple fourth audio recordings. The model training system 110 can also train a base model based on the multiple third text recordings and the global voiceprint features to obtain the first model.

[0122] In some cases, determining global voiceprint features involves extracting multiple third voiceprint features from multiple fourth audio frequencies. The model training system 110 can also obtain aggregated voiceprint features by aggregating multiple third voiceprint features. Furthermore, the model training system 110 can obtain global voiceprint features by normalizing the aggregated voiceprint features. In this way, the model training system 110 can aggregate multiple sentence-by-sentence voiceprint features into a stable object-level representation. Normalization limits the feature scale, thereby improving the numerical stability of the voiceprint conditions and reducing fluctuations in subsequent model processing.

[0123] In some cases, multiple third voiceprint features are aggregated, including performing an arithmetic average on the multiple third voiceprint features to obtain aggregated voiceprint features.

[0124] In some cases, training the base model involves replacing the third voiceprint feature corresponding to each of the multiple third training samples with the global voiceprint feature. The model training system 110 can also train the base model based on multiple third texts and their corresponding global voiceprint features to obtain a first model. In this way, each third training sample can use the global voiceprint features of the same speaker to train the base model. Uniform conditions reduce the variation in voiceprint input between training samples, thereby reducing the overhead of repeated voiceprint extraction and condition preparation within batches.

[0125] In some cases, training the model includes determining, based on a first audio recording, the speech rate features corresponding to the first audio recording and a discrete representation of the target speech. The model training system 110 can also input the first text, target speaker features, and speech rate features into the model to obtain a predicted discrete speech representation. The model training system 110 can also update the model's parameters based on the difference between the predicted discrete speech representation and the target discrete speech representation.

[0126] In some cases, the model's input sequence may include a text portion corresponding to the first text and a speech portion corresponding to the first audio, with the difference determined based on the speech portion. In this way, the model can use the text portion as conditional input and the speech portion to determine the training differences. Loss calculation is concentrated on the speech portion, thereby reducing the number of irrelevant sequence positions involved in gradient calculation.

[0127] In some cases, the model training system 110 may replace the target voiceprint feature with a first zero vector having the same dimension as the target voiceprint feature based on a first discard probability. The model training system 110 may also replace the speech rate feature with a second zero vector having the same dimension as the speech rate feature based on a second discard probability.

[0128] In some cases, the model training system 110 can acquire target text and reference audio, where the reference audio corresponds to the target speaker and the target text corresponds to the target language. The model training system 110 can also determine the third and fourth voiceprint features corresponding to the target speaker based on the reference audio, where the fourth voiceprint feature is the corresponding global voiceprint feature of the target speaker. The model training system 110 can also determine voiceprint conditional features based on at least one of the third and fourth voiceprint features. The model training system 110 can also generate target audio using the model based on the target text and voiceprint conditional features, where the target audio corresponds to the target speaker and the target language.

[0129] In some cases, voiceprint conditional features may include at least one of the following: a third voiceprint feature; a fourth voiceprint feature; or a fused voiceprint feature. The fused voiceprint feature is obtained by fusing the third and fourth voiceprint features based on a first control parameter. The first control parameter determines the fusion ratio of the third and fourth voiceprint features in the fused voiceprint feature, thereby adjusting the degree of accent preservation in the target audio. In this way, voiceprint conditions can be selected among sentence-by-sentence voiceprints, global voiceprints, and fused voiceprints, and the degree of accent preservation in the target audio can be adjusted. The fusion process operates only on a fixed-dimensional vector, thus avoiding the need to store and load multiple models separately for different accent intensities.

[0130] A corresponding apparatus for implementing the above methods or processes is also provided.

[0131] Figure 4 A schematic block diagram of an example device 400 for model training under certain conditions is shown. Device 400 can be implemented as or included in model training system 110. The various modules in device 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0132] like Figure 4 As shown, the device 400 includes an acquisition module 410, a feature determination module 420, a feature selection module 430, and a model training module 440. The acquisition module 410 is configured to acquire multiple training samples. The multiple training samples include a first training sample and at least one second training sample. The first training sample includes a first audio and a first text, and the at least one second training sample includes at least one second audio. The first audio and at least one second audio correspond to a first voice-producing object, the first audio corresponds to a first language, and the at least one second audio corresponds to at least one second language. The first language is different from the second language. The feature determination module 420 is configured to determine a first voiceprint feature and at least one second voiceprint feature based on the multiple training samples. The first voiceprint feature corresponds to the first audio, and the at least one second voiceprint feature corresponds to at least one second audio. The feature selection module 430 is configured to select a target voiceprint feature from the first voiceprint feature or the at least one second voiceprint feature based on a predetermined sampling strategy. The model training module 440 is configured to train a model based on the first text and the target voiceprint feature. The model is configured to generate a third audio of the first voice-producing object based on the first text and the target voiceprint feature, and the third audio corresponds to the first language.

[0133] In some cases, a predetermined sampling strategy can indicate a first probability and a second probability. The feature selection module 430 can be configured to select a target voiceprint feature from at least one second voiceprint feature based on the first probability, and select a first voiceprint feature as the target voiceprint feature based on the second probability.

[0134] In some cases, the first probability may fall within a predetermined probability range, and the second probability may decrease as the first probability increases.

[0135] In some cases, at least one second language may include multiple second languages, and at least one second voiceprint feature may include multiple second voiceprint features corresponding to the multiple second languages ​​respectively. The feature selection module 430 can be configured to determine a first probability corresponding to each second language based on the similarity between the first language and each of the multiple second languages, and select a target voiceprint feature from the multiple second voiceprint features based on the first probability corresponding to each second language.

[0136] In some cases, the feature selection module 430 can be configured to adjust the first probability based on the training progress, so that the first probability increases as the training progresses.

[0137] In some cases, at least one second audio file can be obtained through speech generation processing. The acquisition module 410 can be configured to acquire at least one second text file, which corresponds to at least one second language. The model training module 440 can be configured to generate at least one second audio file using a first model based on global voiceprint features corresponding to the first speaker and at least one second text file. The global voiceprint features are determined based on multiple voiceprint features corresponding to the first speaker.

[0138] In some cases, the first model can be configured to generate discrete speech representations based on text and global speaker features. The model training module 440 can be configured to input at least one second text and global speaker features into the first model to obtain at least one discrete speech representation corresponding to at least one second language, and to convert the at least one discrete speech representation into at least one second audio.

[0139] In some cases, the first model can be obtained by further training the base model. The acquisition module 410 can be configured to acquire multiple third training samples. These multiple third training samples include multiple fourth audio recordings and multiple third text recordings, with the fourth audio recordings corresponding to the first speaker. The feature determination module 420 can be configured to determine global speaker features based on the multiple fourth audio recordings. The model training module 440 can be configured to train the base model based on the multiple third text recordings and the global speaker features to obtain the first model.

[0140] In some cases, the feature determination module 420 can be configured to extract multiple third voiceprint features from multiple fourth audios, obtain aggregated voiceprint features by aggregating the multiple third voiceprint features, and obtain global voiceprint features by normalizing the aggregated voiceprint features.

[0141] In some cases, the feature determination module 420 can be configured to perform an arithmetic average of multiple third voiceprint features to obtain aggregated voiceprint features.

[0142] In some cases, each of the multiple third training samples can have its corresponding third voiceprint feature. The model training module 440 can be configured to replace the third voiceprint feature corresponding to each third training sample with a global voiceprint feature, and train a base model based on multiple third texts and the corresponding global voiceprint features to obtain a first model.

[0143] In some cases, the first audio may correspond to speech rate features and a discrete representation of the target speech. The model training module 440 can be configured to determine the speech rate features and the discrete representation of the target speech based on the first audio, input the first text, target speaker features, and speech rate features into the model to obtain the predicted discrete speech representation, and update the model parameters based on the difference between the predicted discrete speech representation and the target discrete speech representation.

[0144] In some cases, the input sequence of the model may include a text portion corresponding to the first text and a speech portion corresponding to the first audio. The model training module 440 may be configured to determine the difference between the predicted discrete speech representation and the target discrete speech representation based on the speech portion.

[0145] In some cases, a first discard probability can be set for the target voiceprint feature, and a second discard probability can be set for the speech rate feature. The model training module 440 can be configured to replace the target voiceprint feature with a first zero vector with the same dimension as the target voiceprint feature based on the first discard probability, and replace the speech rate feature with a second zero vector with the same dimension as the speech rate feature based on the second discard probability.

[0146] In some cases, device 400 can also be used for application processing of the trained model. Acquisition module 410 can be configured to acquire target text and reference audio. The reference audio corresponds to the target speaker, and the target text corresponds to the target language. Feature determination module 420 can be configured to determine a third and a fourth voiceprint feature corresponding to the target speaker based on the reference audio, wherein the fourth voiceprint feature is a global voiceprint feature corresponding to the target speaker. Feature selection module 430 can be configured to determine voiceprint conditional features based on at least one of the third and fourth voiceprint features. Model training module 440 can also be configured to generate target audio using a model based on the target text and voiceprint conditional features. The target audio corresponds to the target speaker and the target language.

[0147] In some cases, the voiceprint conditional features may include at least one of a third voiceprint feature, a fourth voiceprint feature, or a fused voiceprint feature. The fused voiceprint feature can be obtained by fusing the third and fourth voiceprint features based on a first control parameter. The first control parameter is used to determine the fusion ratio of the third and fourth voiceprint features in the fused voiceprint feature to adjust the degree of accent preservation in the target audio. The feature selection module 430 can be configured to determine the fused voiceprint feature based on the first control parameter and identify the fused voiceprint feature as the voiceprint conditional feature.

[0148] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0149] Figure 5 A block diagram of an electronic device 500 in which one or more examples can be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 5 The electronic device 500 shown can be used to implement the model training system 110, the model application system 120, and the device 400 as discussed above.

[0150] like Figure 5As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0151] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0152] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various examples.

[0153] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0154] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0155] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0156] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0157] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0158] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0159] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0160] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used in this disclosure is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A model training method, comprising: Multiple training samples are obtained, including a first training sample and at least one second training sample. The first training sample includes a first audio and a first text. The at least one second training sample includes at least one second audio. The first audio and at least one second audio correspond to a first vocal object. The first audio corresponds to a first language. The at least one second audio corresponds to at least one second language. The first language is different from the second language. Based on the multiple training samples, a first voiceprint feature and at least one second voiceprint feature are determined, wherein the first voiceprint feature corresponds to the first audio, and the at least one second voiceprint feature corresponds to the at least one second audio. Based on a predetermined sampling strategy, a target voiceprint feature is selected from the first voiceprint feature or the at least one second voiceprint feature; as well as The model is trained based on the first text and the target voiceprint features, and the model is configured to generate a third audio of the first voice object based on the first text and the target voiceprint features, the third audio corresponding to the first language.

2. The method of claim 1, wherein the predetermined sampling strategy indicates a first probability and a second probability, and selecting the target voiceprint feature comprises: Based on the first probability, the target voiceprint feature is selected from the at least one second voiceprint feature; as well as Based on the second probability, the first voiceprint feature is selected as the target voiceprint feature.

3. The method according to claim 2, wherein the first probability is within a predetermined probability range, and the second probability decreases as the first probability increases.

4. The method of claim 2, wherein the at least one second language comprises a plurality of second languages, the at least one second voiceprint feature comprises a plurality of second voiceprint features respectively corresponding to the plurality of second languages, and selecting the target voiceprint feature from the at least one second voiceprint feature comprises: Based on the similarity between the first language and each of the plurality of second languages, the first probability corresponding to each of the second languages ​​is determined; as well as Based on the first probability corresponding to each of the second languages, the target voiceprint feature is selected from the plurality of second voiceprint features.

5. The method according to claim 2, further comprising: The first probability is adjusted based on the training progress so that the first probability increases as the training progresses.

6. The method of claim 1, wherein the at least one second audio signal is generated by the following steps: Obtain at least one second text, the at least one second text corresponding to the at least one second language; and Based on the global voiceprint features corresponding to the first voice-generating object and the at least one second text, the at least one second audio is generated using a first model, wherein the global voiceprint features are determined based on multiple voiceprint features corresponding to the first voice-generating object.

7. The method of claim 6, wherein generating the at least one second audio signal via the first model comprises: The at least one second text and the global voiceprint features are input into the first model to obtain at least one discrete speech representation corresponding to the at least one second language; as well as The at least one discrete speech representation is converted into the at least one second audio.

8. The method of claim 6, wherein the first model is obtained by training the following steps: Acquire multiple third training samples, the multiple third training samples including multiple fourth audios and multiple third texts, the multiple fourth audios corresponding to the first vocal object; Based on the multiple fourth audio frequencies, the global voiceprint features are determined; as well as The first model is obtained by training a base model based on the multiple third texts and the global voiceprint features.

9. The method of claim 8, wherein determining the global voiceprint features comprises: Extract multiple third voiceprint features from the multiple fourth audio frequencies; By aggregating the multiple third voiceprint features, an aggregated voiceprint feature is obtained; as well as The global voiceprint features are obtained by normalizing the aggregated voiceprint features.

10. The method of claim 9, wherein aggregating the plurality of third voiceprint features includes: An arithmetic mean is applied to the plurality of third voiceprint features to obtain the aggregated voiceprint features.

11. The method of claim 9, wherein training the base model comprises: For each of the plurality of third training samples, the third voiceprint feature corresponding to the third training sample is replaced with the global voiceprint feature; as well as The base model is trained based on the multiple third texts and the corresponding global voiceprint features to obtain the first model.

12. The method of claim 1, wherein training the model comprises: Based on the first audio, determine the speech rate features and discrete representation of the target speech corresponding to the first audio; The first text, the target voiceprint features, and the speech rate features are input into the model to obtain a discrete representation of the predicted speech. as well as The parameters of the model are updated based on the difference between the predicted discrete speech representation and the target discrete speech representation.

13. The method of claim 12, wherein the input sequence of the model includes a text portion corresponding to the first text and a speech portion corresponding to the first audio, the difference being determined based on the speech portion.

14. The method of claim 12, further comprising: Based on the first discard probability, the target voiceprint feature is replaced with a first zero vector having the same dimension as the target voiceprint feature; as well as Based on the second discard probability, the speech rate feature is replaced with a second zero vector having the same dimension as the speech rate feature.

15. The method according to claim 1, further comprising: Obtain the target text and reference audio, wherein the reference audio corresponds to the target speaker and the target text corresponds to the target language; Based on the reference audio, the third and fourth voiceprint features corresponding to the target voice source are determined, wherein the fourth voiceprint feature is the global voiceprint feature corresponding to the target voice source. Based on at least one of the third and fourth voiceprint features, determine the voiceprint condition features; as well as Based on the target text and the voiceprint condition features, the model is used to generate target audio, which corresponds to the target vocal object and the target language.

16. The method of claim 15, wherein the voiceprint condition features include at least one of the following: The third voiceprint feature; The fourth voiceprint feature; or The voiceprint features are fused, which are obtained by fusing the third voiceprint feature and the fourth voiceprint feature based on a first control parameter. The first control parameter is used to determine the fusion ratio of the third voiceprint feature and the fourth voiceprint feature in the fused voiceprint feature, so as to adjust the degree of accent preservation in the target audio.

17. An apparatus for model training, comprising: The acquisition module is configured to acquire multiple training samples, the multiple training samples including a first training sample and at least one second training sample, the first training sample including a first audio and a first text, the at least one second training sample including at least one second audio, the first audio and the at least one second audio corresponding to a first vocal object, the first audio corresponding to a first language, the at least one second audio corresponding to at least one second language, and the first language being different from the at least one second language; The feature determination module is configured to determine a first voiceprint feature and at least one second voiceprint feature based on the plurality of training samples, wherein the first voiceprint feature corresponds to the first audio and the at least one second voiceprint feature corresponds to the at least one second audio. The feature selection module is configured to select a target voiceprint feature from the first voiceprint feature or the at least one second voiceprint feature based on a predetermined sampling strategy. as well as The model training module is configured to train a model based on the first text and the target voiceprint features. The model is configured to generate a third audio of the first voice object based on the first text and the target voiceprint features. The third audio corresponds to the first language.

18. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 16 when executed by the at least one processor.

19. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 16.

20. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 16.

Images