Sound conversion model training method, sound conversion method, and device

Abstract

The application provides a sound conversion model training method, a sound conversion method and device, and relates to the technical field of data processing. The method comprises the following steps: obtaining a plurality of different sounder audio samples of sounders; wherein each sounder audio sample corresponds to a fundamental frequency label; training a preset sound conversion model according to the automatic speech recognition sample features and the sounder sample feature vectors corresponding to each sounder audio sample, to obtain a first sound conversion model; training the first sound conversion model based on the counterfeit automatic speech recognition features and the sounder sample feature vectors corresponding to each sounder audio counterfeit sample of a target sounder, to obtain a second sound conversion model; wherein the sounder audio counterfeit sample is obtained according to the real sounder audio sample of the target sounder; and the second sound conversion model is used to output the fundamental frequency information corresponding to the sounder audio based on the automatic speech recognition features and the sounder feature vectors corresponding to the sounder audio.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for training a voice conversion model, a voice conversion method, and an apparatus. Background Technology

[0002] Voice conversion technology, a field within speech synthesis, is capable of transforming one person's voice into the timbre of another designated speaker. This technology has wide applications in various areas, including synthesizing audio and video dubbing for specific speakers, and music entertainment.

[0003] In related technologies, although the voice conversion model can restore the fundamental frequency information of a specific speaker to a certain extent, and thus generate audio with the voice characteristics of a specific speaker based on the fundamental frequency information, there is still considerable room for improvement in the performance of the voice conversion model in related technologies.

[0004] Therefore, how to generate more efficient voice conversion models has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] This invention provides a method for training a voice conversion model, a voice conversion method, and an apparatus to solve the problem of how to generate a more efficient voice conversion model in the prior art.

[0006] This invention provides a method for training a sound conversion model, comprising:

[0007] Acquire audio samples from multiple different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency tag;

[0008] Based on the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the speaker audio samples, a preset voice conversion model is trained to obtain a first voice conversion model;

[0009] Based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each speaker's audio forgery samples of the target speaker, the first voice conversion model is trained to obtain the second voice conversion model;

[0010] The speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0011] According to a voice conversion model training method provided by the present invention, the method for generating the speaker audio forgery sample includes:

[0012] Obtain all audio samples of the target speaker and multiple sets of random number seeds;

[0013] Based on multiple sets of random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

[0014] According to a voice conversion model training method provided by the present invention, the step of performing pitch and frequency band transformations on each of the speaker audio samples based on multiple sets of random number seeds to obtain multiple speaker audio forgery samples with adjusted timbre includes:

[0015] For any set of random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch transformation is performed on each speaker's audio sample to obtain each speaker's audio sample after pitch transformation.

[0016] Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker audio fake sample with adjusted timbre.

[0017] By iterating through each of the random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

[0018] According to a voice conversion model training method provided by the present invention, the step of training a preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker audio samples to obtain a first voice conversion model includes:

[0019] Each speaker's audio sample is used as a first training sample, along with the corresponding automatic speech recognition sample features, speaker sample feature vector, and fundamental frequency label, to obtain multiple first training samples.

[0020] For any first training sample, the first training sample is input into the preset sound conversion model, and the fundamental frequency information corresponding to the first training sample is output.

[0021] Using a first preset loss function, a first loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein, the first preset loss function includes: a reconstruction loss function and an adversarial loss function;

[0022] If the first loss value is less than the first preset threshold, training is stopped, and the first voice conversion model is obtained.

[0023] According to a voice conversion model training method provided by the present invention, the first voice conversion model is trained based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each forged audio sample of the target speaker to obtain a second voice conversion model, comprising:

[0024] Each of the speaker audio forgery samples corresponds to a forged automatic speech recognition sample feature, a speaker sample feature vector, and a fundamental frequency label as a second training sample, and multiple second training samples are obtained.

[0025] For any second training sample, input the second training sample into the first sound conversion model and output the fundamental frequency information corresponding to the second training sample;

[0026] Using a second preset loss function, a second loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the second training sample; wherein, the second preset loss function only includes the reconstruction loss function;

[0027] If the second loss value is less than the second preset threshold, training is stopped, and the second voice conversion model is obtained.

[0028] The present invention also provides a sound conversion method, comprising:

[0029] Automatic speech recognition features and speaker feature vectors are obtained from the audio of the target speaker.

[0030] The automatic speech recognition features and the speaker feature vector are input into the second voice conversion model, and the fundamental frequency information of the target speaker's audio is output.

[0031] The second voice conversion model is trained based on the voice conversion model training method described in any of the above items.

[0032] According to the voice conversion method provided by the present invention, after the step of inputting the automatic speech recognition features and the speaker feature vector into a second voice conversion model and outputting the fundamental frequency information of the target speaker's audio, the method further includes:

[0033] The fundamental frequency information of the target speaker's audio is converted using a vocoder to obtain the converted audio.

[0034] The present invention also provides a sound conversion model training device, comprising:

[0035] The first acquisition module is used to acquire audio samples of multiple different speakers; wherein each speaker audio sample corresponds to a fundamental frequency tag;

[0036] The first training module is used to train the preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker audio samples, so as to obtain the first voice conversion model.

[0037] The second training module is used to train the first voice conversion model based on the fake automatic speech recognition features and speaker sample feature vectors corresponding to each fake speaker audio sample of the target speaker, so as to obtain the second voice conversion model.

[0038] The speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0039] The present invention also provides a sound conversion device, comprising:

[0040] The second acquisition module is used to acquire the automatic speech recognition features and speaker feature vector of the target speaker's audio.

[0041] The conversion module is used to input the automatic speech recognition features and the speaker feature vector into the second voice conversion model and output the fundamental frequency information of the target speaker's audio.

[0042] The second voice conversion model is trained based on the voice conversion model training method described in any of the above items.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sound conversion model training method or the sound conversion method described above.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sound conversion model training method or the sound conversion method as described above.

[0045] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the sound conversion model training method or the sound conversion method as described above.

[0046] The present invention provides a voice conversion model training method, voice conversion method, and apparatus. By collecting audio samples from multiple different speakers, a diverse training dataset can be constructed, thereby training a first voice conversion model capable of handling multiple voice features. Then, the first voice conversion model can be further fine-tuned using forged audio samples from the target speaker to obtain a second voice conversion model. This allows the model to learn the target speaker's voice features more precisely. The second voice conversion model can learn the target speaker's voice features more precisely, improving the similarity of timbre conversion and effectively reducing timbre leakage. The second voice conversion model is more closely aligned with the target speaker's voice characteristics, improving the personalization and similarity of the voice conversion. Finally, the trained second voice conversion model can output the fundamental frequency information corresponding to the target speaker's audio based on the input automatic speech recognition features and speaker feature vectors, thereby achieving high-quality timbre conversion and addressing the timbre leakage problem caused by training with multiple audio sources. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the sound conversion model training method provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the forged data generation process provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the sound conversion method provided by the present invention;

[0051] Figure 4 A complete flowchart of sound conversion provided for this invention;

[0052] Figure 5 The structural diagram of the sound conversion model provided by this invention;

[0053] Figure 6 This is a schematic diagram of the sound conversion model training device provided by the present invention;

[0054] Figure 7 A schematic diagram of the sound conversion device provided by the present invention;

[0055] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] In related technologies, traditional voice conversion algorithms are typically trained based on a reconstruction framework, meaning that the target speaker's audio is used during training. This approach results in the model learning from the same speaker for both input and output data, which is incompatible with the requirement for cross-speaker feature conversion in real-world voice conversion tasks. Therefore, in practical applications, voice conversion algorithms may not be able to effectively and completely convert the source audio into the target speaker's audio, leading to timbre leakage from the source audio.

[0058] Figure 1 This is a flowchart illustrating the sound conversion model training method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0059] Step 110: Obtain audio samples from multiple different speakers; wherein each speaker audio sample corresponds to a fundamental frequency tag;

[0060] In this invention, by acquiring audio samples from multiple different speakers, a dataset containing diverse voice features can be constructed. This dataset can be used to train a voice conversion model that can understand and imitate the features of different speakers.

[0061] In this invention, by collecting audio samples from different speakers, the model can learn the common features and individual differences of various voices, thereby improving its generalization ability.

[0062] In an optional embodiment, audio samples of speakers with more than 200 speakers and a single audio duration of more than 1 hour can be obtained. When obtaining speaker audio samples, audio samples of various styles can be obtained.

[0063] More specifically, in this invention, each audio sample is equipped with a fundamental frequency label, enabling the model to not only learn the timbre and rhythmic features of the sound during training, but also accurately capture and convert the pitch information of the sound.

[0064] Step 120: Train the preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker audio samples to obtain the first voice conversion model;

[0065] In this invention, after denoising the audio samples of the speaker, the continuous audio signal is segmented into shorter frames for local analysis. Then, acoustic features such as the Mel-frequency cepstral coefficients of each frame are calculated, and spectral information of the speech signal is captured as spectral information for automatic speech recognition. Techniques such as i-vector or x-vector can also be used to extract speaker sample feature vectors that characterize the speaker's identity.

[0066] In this invention, a preset voice conversion model is trained, and the model learns how to map the input automatic speech recognition features and speaker feature vectors to the target output, namely the fundamental frequency information of the target speaker. After training, the resulting first voice conversion model can process any input speaker audio sample to ensure that the first voice conversion model has basic reconstruction capabilities. In other words, the first voice conversion model in this application has the ability to process audio from different speakers, thus laying the groundwork for subsequent secondary training.

[0067] Step 130: Based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each speaker audio forgery sample of the target speaker, train the first voice conversion model to obtain the second voice conversion model;

[0068] The speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0069] In this invention, the target speaker can specifically refer to any one of multiple different speakers, and it can specifically be a speaker specified by the user.

[0070] In this invention, based on the real voice audio sample of the target voice speaker, a fake voice audio sample is generated by means of fundamental frequency transformation, and the fake automatic speech recognition features of the fake voice audio sample are obtained.

[0071] These fake samples retain the features of the target speaker, as well as the fundamental frequency label and speaker feature vector corresponding to the real speaker's audio sample. However, the timbre differs from the original audio, providing the model with more diverse training data.

[0072] In this invention, the first voice conversion model, which has been initially trained, is fine-tuned using these forged samples to obtain a second voice conversion model. This fine-tuning process allows the model to learn the vocal characteristics of the target speaker more precisely, improving the similarity of the converted timbre.

[0073] In this invention, since the second voice conversion model is obtained by retraining on the basis of the first voice conversion model, the second voice conversion model itself has the ability to convert voices of various different human voices. After secondary training, it can effectively use parallel datasets of audio with different timbre (but the same content) from the target speaker to make the data properties consistent during the training and inference stages, thereby improving the voice conversion effect and reducing the timbre leakage problem.

[0074] In this invention, the second voice conversion model can output fundamental frequency information corresponding to the target speaker's audio based on the input automatic speech recognition features and speaker feature vectors. This generates converted speech that more closely resembles the target speaker's natural voice. This technology can be applied to fields such as speech synthesis, video dubbing, and speech recognition, providing higher-quality voice conversion services and enhancing user experience and application usability.

[0075] In this invention, by collecting audio samples from multiple different speakers, a diverse training dataset can be constructed, thereby training a first voice conversion model capable of handling various voice features. Then, it can be further fine-tuned using forged audio samples of the target speaker, allowing the model to learn the target speaker's voice features more precisely, resulting in a second voice conversion model. During the training of the second voice conversion model, parallel datasets are constructed by generating audio with different timbres (but identical content) than the target speaker, ensuring consistency in data properties between the training and inference phases. This improves the voice conversion effect, reduces timbre leakage, and the trained second voice conversion model more closely matches the target speaker's voice characteristics, enhancing the personalization and similarity of the voice conversion. Finally, the trained second voice conversion model can output fundamental frequency information corresponding to the target speaker's audio based on the input automatic speech recognition features and speaker feature vectors, thus achieving high-quality timbre conversion.

[0076] Optionally, the method for generating the speaker audio forgery sample includes:

[0077] Obtain all audio samples of the target speaker and multiple sets of random number seeds;

[0078] Based on multiple sets of random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

[0079] In this invention, different random number seeds can be pre-designed. A random number seed is a component in an algorithm or program used to generate random numbers. Specifically, it can be used to generate random pitch amplitude adjustment information and random frequency band change adjustment information.

[0080] Optionally, the step of performing pitch and frequency band transformations on each of the speaker audio samples according to multiple sets of the random number seeds to obtain multiple speaker audio forgery samples with adjusted timbre includes:

[0081] For any set of random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch transformation is performed on each speaker's audio sample to obtain each speaker's audio sample after pitch transformation.

[0082] Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker audio fake sample with adjusted timbre.

[0083] By iterating through each of the random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

[0084] In this invention, for each set of random number seeds, the pitch of each speaker's audio sample is transformed based on the random pitch amplitude adjustment information generated by the seed. This means changing the fundamental frequency, i.e., the pitch, of the audio sample to simulate the pitch characteristics of different speakers.

[0085] Similarly, based on each set of random number seeds, the frequency bands of the audio samples are transformed using the random frequency band change adjustment information generated by the seeds. This involves adjusting the spectral envelope of the audio signal to change the timbre and simulate the resonance characteristics of different speakers.

[0086] Frequency band switching can be achieved by changing the parameters of the filter, such as adjusting the center frequency of a formal filter bank, to simulate the oral cavity and vocal tract characteristics of different speakers.

[0087] In this invention, all random number seeds are traversed, and the pitch and frequency band transformation processes described above are repeated multiple times for each speaker's audio sample. This generates multiple timbre-adjusted speaker audio forgery samples, each corresponding to a different random number seed, thereby increasing the diversity of the samples.

[0088] In this invention, the timbre of each speaker's audio sample is adjusted through pitch and frequency band transformation, generating timbre-adjusted speaker audio forgery samples. These forgery samples are consistent with the original audio samples in content, but differ in timbre, thus providing diverse training data.

[0089] In one alternative embodiment, Figure 2 This is a schematic diagram of the forged data generation process provided by the present invention, such as... Figure 2 As shown, each time data forgery is performed, a set of random number seeds is designed and used as the base frequency conversion parameter.

[0090] Extract the fundamental frequency of the target speaker's audio and determine the magnitude of pitch increase and decrease based on the set random number seed to initially change the timbre of the source audio.

[0091] Then, by using low-shelf filter, high-shelf filter, and peak filter, the frequency bands of the audio are subjected to random gain or attenuation based on random parameters, thereby further changing the timbre of the source audio.

[0092] Optionally, the step of training a preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker's audio samples to obtain a first voice conversion model includes:

[0093] Each speaker's audio sample is used as a first training sample, along with the corresponding automatic speech recognition sample features, speaker sample feature vector, and fundamental frequency label, to obtain multiple first training samples.

[0094] For any first training sample, the first training sample is input into the preset sound conversion model, and the fundamental frequency information corresponding to the first training sample is output.

[0095] Using a first preset loss function, a first loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein, the first preset loss function includes: a reconstruction loss function and an adversarial loss function;

[0096] If the first loss value is less than the first preset threshold, training is stopped, and the first voice conversion model is obtained.

[0097] In this invention, the automatic speech recognition features, speaker sample feature vectors, and fundamental frequency labels corresponding to each speaker's audio sample are combined into a first training sample.

[0098] In this way, multiple first training samples are prepared for model training, each containing complete input features and target output labels.

[0099] In this invention, for any given first training sample, it is input into a preset voice conversion model. The model outputs the corresponding fundamental frequency information based on the input automatic speech recognition features and the speaker feature vector. Using a first preset loss function, a loss value is calculated based on the fundamental frequency information output by the model and the fundamental frequency label corresponding to the training sample.

[0100] The first preset loss function consists of two parts: the reconstruction loss function and the adversarial loss function.

[0101] Reconstruction loss function: measures the difference between the model output and the true fundamental frequency label, ensuring that the model can accurately reconstruct the target fundamental frequency; Adversarial loss function: used in generative adversarial networks to measure the discriminative power between the generated fundamental frequency information and the true fundamental frequency information, driving the model to generate more realistic fundamental frequency information.

[0102] When the calculated initial loss value is less than a preset threshold, the model is considered to have achieved a certain training effect, and training is stopped at this point. This threshold is a pre-set performance metric used to determine when to stop training to avoid overfitting and ensure that the model has good generalization ability.

[0103] After training, the resulting first voice conversion model can accurately output fundamental frequency information based on the input automatic speech recognition features and the speaker feature vector, which can be used for subsequent voice conversion tasks.

[0104] Optionally, the step of training the first voice conversion model based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each speaker's audio forgery samples of the target speaker to obtain the second voice conversion model includes:

[0105] Each of the speaker audio forgery samples corresponds to a forged automatic speech recognition sample feature, a speaker sample feature vector, and a fundamental frequency label as a second training sample, and multiple second training samples are obtained.

[0106] For any second training sample, input the second training sample into the first sound conversion model and output the fundamental frequency information corresponding to the second training sample;

[0107] Using a second preset loss function, a second loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the second training sample; wherein, the second preset loss function only includes the reconstruction loss function;

[0108] If the second loss value is less than the second preset threshold, training is stopped, and the second voice conversion model is obtained.

[0109] In this invention, the forged automatic speech recognition features, the speaker sample feature vector, and the fundamental frequency label corresponding to each speaker audio forgery sample are combined into a second training sample. In this way, multiple second training samples are prepared for fine-tuning the model, and each sample contains forged input features and target output labels.

[0110] For any second training sample, it is input into the first voice conversion model. The model outputs the corresponding fundamental frequency information based on the input forged automatic speech recognition sample and the speaker's feature vector. Using a second preset loss function, a second loss value is calculated based on the fundamental frequency information output by the model and the fundamental frequency label corresponding to the second training sample.

[0111] The second preset loss function only includes the reconstruction loss function. The model training focuses only on minimizing the difference between the output fundamental frequency and the true fundamental frequency label, without involving the adversarial loss function. This is because the overall loss may increase significantly in the later stages of model training, manifesting as obvious errors in the pronunciation of the synthesized audio. Further investigation revealed that the gradient loss originated from the learning collapse of the discriminator. Since the voice conversion task is not a reconstruction task, this invention attempts to remove the discriminator, that is, to remove the adversarial loss while retaining the reconstruction loss. This approach allows the training mode based on the fundamental frequency fake data to train to more steps without gradient explosion, improving audio similarity without sacrificing the pronunciation of the synthesized audio.

[0112] In this invention, when the calculated second loss value is less than a preset second threshold, the model is considered to have achieved a certain training effect, and training is stopped at this point. After training, the resulting second voice conversion model can accurately output fundamental frequency information based on the input forged automatic speech recognition features and speaker feature vectors, which can be used for subsequent voice conversion tasks, especially in simulating the voice features of the target speaker.

[0113] In this invention, the training process of the second voice conversion model fine-tunes the first voice conversion model using fake samples, further optimizing the model's performance and making it better at simulating the voice characteristics of the target speaker. Furthermore, by using only the reconstruction loss function, the model's training focuses more on improving the accuracy of voice conversion rather than generating realism in adversarial training, thus avoiding the gradient explosion problem in the later stages of training.

[0114] Figure 3 This is a schematic diagram of the sound conversion method provided by the present invention, such as... Figure 3 As shown, it includes:

[0115] Step 310: Obtain the automatic speech recognition features and speaker feature vector of the target speaker's audio.

[0116] In this invention, automatic speech recognition features are acoustic features extracted from speech signals to characterize the properties of the speech signals for speech recognition. The most common automatic speech recognition feature is the Mel-frequency cepstral coefficient. The Mel-frequency cepstral coefficient is obtained by converting the speech signal into a spectrum on the Mel-frequency scale.

[0117] In this invention, the speaker feature vector is a feature used to characterize the speaker's identity information. In the field of speaker recognition, the X-vector is a widely used speaker feature vector.

[0118] Step 320: Input the automatic speech recognition features and the speaker feature vector into the second voice conversion model, and output the fundamental frequency information of the target speaker's audio.

[0119] The second voice conversion model is trained based on the voice conversion model training method described in any of the above items.

[0120] In this invention, these features are fed as input data into a second voice conversion model. This model is obtained by performing secondary training on the first voice conversion model in the above steps, and it has learned how to generate the fundamental frequency information of the target speaker based on the input features.

[0121] In this invention, the second voice conversion model receives automatic speech recognition features and a speaker feature vector. Through its internal neural network structure, it processes these features and learns the relationship between them and the fundamental frequency of the target speaker. The model's output is fundamental frequency information, i.e., a representation of pitch or pitch variation. This output is crucial in the voice conversion process because it determines the pitch characteristics of the converted audio, thus affecting the perceived similarity of the voice. This fundamental frequency information can effectively adjust the input audio to a timbre similar to the target speaker.

[0122] In this invention, the output fundamental frequency information can be used for further sound synthesis or sound conversion processes, such as synthesizing the fundamental frequency information and other acoustic features into an audible audio signal using a vocoder, or adjusting the pitch of existing audio to match the voice of the target speaker.

[0123] In one alternative embodiment, Figure 4 The complete flowchart of sound conversion provided by this invention is as follows: Figure 4 As shown, it includes:

[0124] First, prepare data from multiple people, requiring different styles; the total number of speakers should be between 100 and 200; the audio duration of each speaker should be more than 1 hour.

[0125] Then, automatic speech recognition feature extraction is performed on the audio data to form the content features of the input audio. Speaker feature vector extraction is performed, and together with the content features, they are used as the input of the decoder. At the same time, the fundamental frequency is extracted and used as the fundamental frequency label, which is used as the optimization target.

[0126] Then, based on the above data, a multi-person voice conversion model is trained using the reconstruction framework;

[0127] For the target speaker, a fundamental frequency spoofing algorithm is used to synthesize 100-200 sets of parallel data, and the spoofed automatic speech recognition features of the parallel data are extracted;

[0128] Based on the pure voice conversion model framework, a single-person voice conversion model is trained on fundamental frequency parallel spoofed data, and the weights of a multi-person model are loaded to obtain the final voice conversion model.

[0129] In one alternative embodiment, Figure 5 The sound conversion model structure diagram provided by this invention is as follows: Figure 5 As shown, the voice conversion process begins by acquiring features from a forged automatic speech recognition source. These features are derived from a baseband forgery source and used as content features. Next, these content features are combined with the feature vectors of the speaker samples from the baseband forgery source to form the input to the model. This input data is fed into the voice conversion model, which then outputs synthesized baseband information.

[0130] The synthesized fundamental frequency information, along with the corresponding fundamental frequency label from the fundamental frequency forgery source, is used to calculate the loss value. The loss calculation can consist of two main parts: reconstruction loss and adversarial loss. The reconstruction loss ensures the consistency between the fundamental frequency information output by the model and the target fundamental frequency label, while the adversarial loss is used to improve the quality and realism of the synthesized fundamental frequency.

[0131] Finally, the synthesized fundamental frequency information is fed into a vocoder, which is responsible for converting this fundamental frequency information into audible synthesized audio. In this way, through the structured design of the model and precise loss calculation, high-quality audio synthesis can be achieved, generating audio output that matches the timbre of the target speaker.

[0132] The apparatus provided by the present invention will now be described, and the apparatus described below can be referred to in correspondence with the method described above.

[0133] Figure 6 This is a schematic diagram of the sound conversion model training device provided by the present invention, as shown below. Figure 6 As shown, it includes:

[0134] The first acquisition module 610 is used to acquire audio samples of multiple different speakers; wherein each speaker audio sample corresponds to a fundamental frequency tag;

[0135] The first training module 620 is used to train a preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker audio samples, so as to obtain the first voice conversion model.

[0136] The second training module 630 is used to train the first voice conversion model based on the fake automatic speech recognition features and speaker sample feature vectors corresponding to each fake speaker audio sample of the target speaker, so as to obtain the second voice conversion model.

[0137] The speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0138] Optionally, the device is further used for:

[0139] Obtain all audio samples of the target speaker and multiple sets of random number seeds;

[0140] Based on multiple sets of random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

[0141] Optionally, the device is further used for:

[0142] For any set of random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch transformation is performed on each speaker's audio sample to obtain each speaker's audio sample after pitch transformation.

[0143] Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker audio fake sample with adjusted timbre.

[0144] By iterating through each of the random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

[0145] Optionally, the device is further used for:

[0146] Each speaker's audio sample is used as a first training sample, along with the corresponding automatic speech recognition sample features, speaker sample feature vector, and fundamental frequency label, to obtain multiple first training samples.

[0147] For any first training sample, the first training sample is input into the preset sound conversion model, and the fundamental frequency information corresponding to the first training sample is output.

[0148] Using a first preset loss function, a first loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein, the first preset loss function includes: a reconstruction loss function and an adversarial loss function;

[0149] If the first loss value is less than the first preset threshold, training is stopped, and the first voice conversion model is obtained.

[0150] Optionally, the device is further used for:

[0151] Each of the speaker audio forgery samples corresponds to a forged automatic speech recognition sample feature, a speaker sample feature vector, and a fundamental frequency label as a second training sample, and multiple second training samples are obtained.

[0152] For any second training sample, input the second training sample into the first sound conversion model and output the fundamental frequency information corresponding to the second training sample;

[0153] Using a second preset loss function, a second loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the second training sample; wherein, the second preset loss function only includes the reconstruction loss function;

[0154] If the second loss value is less than the second preset threshold, training is stopped, and the second voice conversion model is obtained.

[0155] Figure 7 This is a schematic diagram of the sound conversion device provided by the present invention, as shown below. Figure 7 As shown, it includes:

[0156] The second acquisition module 710 is used to acquire the automatic speech recognition features and speaker feature vector of the target speaker's audio.

[0157] The conversion module 720 is used to input the automatic speech recognition features and the speaker feature vector into the second voice conversion model and output the fundamental frequency information of the target speaker audio.

[0158] The second voice conversion model is trained based on the voice conversion model training method described in any of the above items.

[0159] Optionally, the device is further used for:

[0160] The fundamental frequency information of the target speaker's audio is converted using a vocoder to obtain the converted audio.

[0161] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a voice conversion model training method or a voice conversion method. The method includes: acquiring multiple audio samples from different speakers; wherein each speaker audio sample corresponds to a fundamental frequency label; training a preset voice conversion model based on the automatic speech recognition sample features and speaker sample feature vectors corresponding to each speaker audio sample to obtain a first voice conversion model; training the first voice conversion model based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each forged speaker audio sample of the target speaker to obtain a second voice conversion model; wherein the forged speaker audio samples are obtained based on the real speaker audio samples of the target speaker; and the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vectors corresponding to the speaker audio.

[0162] Alternatively, obtain the automatic speech recognition features and speaker feature vector of the target speaker's audio; input the automatic speech recognition features and speaker feature vector into a second voice conversion model, and output the fundamental frequency information of the target speaker's audio; wherein, the second voice conversion model is trained based on the voice conversion model training method described in any of the above embodiments.

[0163] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the voice conversion model training method or voice conversion method provided by the above methods. The method includes: acquiring multiple voice audio samples from different speakers; wherein each voice audio sample corresponds to a fundamental frequency label; training a preset voice conversion model based on the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the voice audio samples to obtain a first voice conversion model; training the first voice conversion model based on the fake automatic speech recognition features and speaker sample feature vectors corresponding to each fake voice audio sample of the target speaker to obtain a second voice conversion model; wherein the fake voice audio samples are obtained based on the real voice audio samples of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the voice audio based on the automatic speech recognition features and speaker feature vectors corresponding to the voice audio.

[0165] Alternatively, obtain the automatic speech recognition features and speaker feature vector of the target speaker's audio; input the automatic speech recognition features and speaker feature vector into a second voice conversion model, and output the fundamental frequency information of the target speaker's audio; wherein, the second voice conversion model is trained based on the voice conversion model training method described in any of the above embodiments.

[0166] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the voice conversion model training method or voice conversion method provided by the above methods. The method includes: acquiring multiple voice audio samples from different speakers; wherein each voice audio sample corresponds to a fundamental frequency label; training a preset voice conversion model based on the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the voice audio samples to obtain a first voice conversion model; training the first voice conversion model based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each forged voice audio sample of the target speaker to obtain a second voice conversion model; wherein the forged voice audio samples are obtained based on the real voice audio samples of the target speaker; and the second voice conversion model is used to output the fundamental frequency information corresponding to the voice audio based on the automatic speech recognition features and speaker feature vectors corresponding to the voice audio.

[0167] Alternatively, obtain the automatic speech recognition features and speaker feature vector of the target speaker's audio; input the automatic speech recognition features and speaker feature vector into a second voice conversion model, and output the fundamental frequency information of the target speaker's audio; wherein, the second voice conversion model is trained based on the voice conversion model training method described in any of the above embodiments.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a voice conversion model, characterized in that, include: Acquire audio samples from multiple different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency tag; Based on the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the speaker audio samples, a preset voice conversion model is trained to obtain a first voice conversion model; Based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each speaker's audio forgery samples of the target speaker, the first voice conversion model is trained to obtain the second voice conversion model; The speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio. The step of training a preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker's audio samples to obtain a first voice conversion model includes: Each speaker's audio sample is used as a first training sample, along with the corresponding automatic speech recognition sample features, speaker sample feature vector, and fundamental frequency label, to obtain multiple first training samples. For any first training sample, the first training sample is input into the preset sound conversion model, and the fundamental frequency information corresponding to the first training sample is output. Using a first preset loss function, a first loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein, the first preset loss function includes: a reconstruction loss function and an adversarial loss function; If the first loss value is less than the first preset threshold, training is stopped, and the first voice conversion model is obtained.

2. The sound conversion model training method according to claim 1, characterized in that, The method for generating the speaker audio forgery sample includes: Obtain all audio samples of the target speaker and multiple sets of random number seeds; Based on multiple sets of random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

3. The sound conversion model training method according to claim 2, characterized in that, The process of performing pitch and frequency band transformations on each of the speaker audio samples based on multiple sets of random number seeds to obtain multiple speaker audio forgery samples with adjusted timbre includes: For any set of random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch transformation is performed on each speaker's audio sample to obtain each speaker's audio sample after pitch transformation. Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker audio fake sample with adjusted timbre. By iterating through each of the random number seeds, pitch and frequency band transformations are performed on each of the speaker audio samples to obtain multiple speaker audio fake samples with adjusted timbre.

4. The sound conversion model training method according to claim 1, characterized in that, The second voice conversion model is obtained by training the first voice conversion model based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each speaker's audio forgery samples of the target speaker, including: Each of the speaker audio forgery samples corresponds to a forged automatic speech recognition sample feature, a speaker sample feature vector, and a fundamental frequency label as a second training sample, and multiple second training samples are obtained. For any second training sample, input the second training sample into the first sound conversion model and output the fundamental frequency information corresponding to the second training sample; Using a second preset loss function, a second loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the second training sample; wherein, the second preset loss function only includes the reconstruction loss function; If the second loss value is less than the second preset threshold, training is stopped, and the second voice conversion model is obtained.

5. A sound conversion method, characterized in that, include: Automatic speech recognition features and speaker feature vectors are obtained from the audio of the target speaker. The automatic speech recognition features and the speaker feature vector are input into the second voice conversion model, and the fundamental frequency information corresponding to the target speaker's audio is output. The second voice conversion model is trained based on the voice conversion model training method described in any one of claims 1-4.

6. The sound conversion method according to claim 5, characterized in that, After the step of inputting the automatic speech recognition features and the speaker feature vector into the second voice conversion model and outputting the fundamental frequency information of the target speaker's audio, the method further includes: The fundamental frequency information of the target speaker's audio is converted using a vocoder to obtain the converted audio.

7. A sound conversion model training device, characterized in that, include: The first acquisition module is used to acquire audio samples of multiple different speakers; wherein each speaker audio sample corresponds to a fundamental frequency tag; The first training module is used to train the preset voice conversion model based on the automatic speech recognition sample features and the speaker sample feature vector corresponding to each of the speaker audio samples, so as to obtain the first voice conversion model. The second training module is used to train the first voice conversion model based on the fake automatic speech recognition features and speaker sample feature vectors corresponding to each fake speaker audio sample of the target speaker, so as to obtain the second voice conversion model. The speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second voice conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio. The device is also used for: Each speaker's audio sample is used as a first training sample, along with the corresponding automatic speech recognition sample features, speaker sample feature vector, and fundamental frequency label, to obtain multiple first training samples. For any first training sample, the first training sample is input into the preset sound conversion model, and the fundamental frequency information corresponding to the first training sample is output. Using a first preset loss function, a first loss value is calculated based on the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein, the first preset loss function includes: a reconstruction loss function and an adversarial loss function; If the first loss value is less than the first preset threshold, training is stopped, and the first voice conversion model is obtained.

8. A sound conversion device, characterized in that, include: The second acquisition module is used to acquire the automatic speech recognition features and speaker feature vector of the target speaker's audio. The conversion module is used to input the automatic speech recognition features and the speaker feature vector into the second voice conversion model and output the fundamental frequency information of the target speaker's audio. The second voice conversion model is trained based on the voice conversion model training method described in any one of claims 1-4.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the voice conversion model training method as described in any one of claims 1 to 4 or the voice conversion method as described in any one of claims 5 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice conversion model training method as described in any one of claims 1 to 4 or the voice conversion method as described in any one of claims 5 to 6.

Images