Sound conversion model training method, sound conversion method and device

Abstract

The invention provides a sound conversion model training method and device, and a sound conversion method and device, and relates to the technical field of data processing, and the method comprises the steps: obtaining sound production person audio samples of a plurality of different sound production persons; wherein each sounder audio sample corresponds to one fundamental frequency label; training a preset sound conversion model according to the automatic speech recognition sample features and the sound producer sample feature vectors corresponding to the sound producer audio samples to obtain a first sound conversion model; training the first sound conversion model based on the forged automatic speech recognition features corresponding to each speaker audio forged sample of the target speaker and the speaker sample feature vector to obtain a second sound conversion model; wherein the sounder audio forged sample is obtained according to a real sounder audio sample of the target sounder; and the second sound conversion model is used for outputting fundamental frequency information corresponding to the voice frequency of the sounder based on the automatic voice recognition features corresponding to the voice frequency of the sounder and the feature vector of the sounder.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a sound conversion model training method, a sound conversion method and a device. Background Art

[0002] Voice conversion technology in the field of speech synthesis can convert a person's voice into the timbre of another specified speaker. Voice conversion technology has a wide range of applications in many fields, such as synthesizing audio of a specific speaker, video dubbing, and music entertainment.

[0003] In the related art, although the sound conversion model can restore the fundamental frequency information of a specific speaker to a certain extent, and thus generate audio with the voice characteristics of the specific speaker based on the fundamental frequency information, there is still much room for improvement in the performance of the sound conversion model in the related art.

[0004] Therefore, how to generate a more efficient sound conversion model has become a problem that needs to be solved urgently in the industry. Summary of the invention

[0005] The present invention provides a sound conversion model training method, a sound conversion method and a device, which are used to solve the problem of how to generate a more efficient sound conversion model in the prior art.

[0006] The present invention provides a sound conversion model training method, comprising: Acquire multiple speaker audio samples of different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; Training a preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the speaker audio samples to obtain a first sound conversion model; Based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector, the first sound conversion model is trained to obtain a second sound conversion model; Among them, the speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0007] According to a sound conversion model training method provided by the present invention, the method for generating a speaker audio forgery sample comprises: Obtaining all audio samples of the target speaker corresponding to the target speaker, and obtaining multiple groups of random number seeds; The pitch and frequency band of each speaker's audio sample are transformed according to the multiple groups of random number seeds to obtain multiple speaker's audio forged samples with adjusted timbre.

[0008] According to a sound conversion model training method provided by the present invention, the pitch conversion and frequency band conversion of each speaker audio sample are performed according to multiple groups of random number seeds to obtain multiple speaker audio forged samples with adjusted timbre, including: For any group of the random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch-transform the audio samples of each speaker to obtain the audio samples of each speaker after pitch transformation; Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker's audio forged sample with adjusted timbre; Each of the random number seeds is traversed, and the pitch and frequency band of each of the speaker's audio samples are transformed to obtain a plurality of speaker audio forgery samples with adjusted timbre.

[0009] According to a sound conversion model training method provided by the present invention, the preset sound conversion model is trained according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the audio samples of each speaker to obtain a first sound conversion model, including: Taking the automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each speaker audio sample as a first training sample, and obtaining a plurality of first training samples; For any first training sample, input the first training sample into the preset sound conversion model, and output fundamental frequency information corresponding to the first training sample; Using a first preset loss function, a first loss value is calculated according to the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein the first preset loss function includes: a reconstruction loss function and an adversarial loss function; When the first loss value is less than a first preset threshold, the training is stopped to obtain a first sound conversion model.

[0010] According to a method for training a voice conversion model provided by the present invention, the method of training the first voice conversion model based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector to obtain the second voice conversion model includes: Taking the forged automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each of the forged audio samples of the speaker as a second training sample, and obtaining a plurality of second training samples; For any second training sample, input the second training sample into the first sound conversion model, and output fundamental frequency information corresponding to the second training sample; Using a second preset loss function, calculating a second loss value according to the fundamental frequency information corresponding to the second training sample and the fundamental frequency label corresponding to the second training sample; wherein the second preset loss function only includes a reconstruction loss function; When the second loss value is less than a second preset threshold, the training is stopped to obtain a second sound conversion model.

[0011] The present invention also provides a sound conversion method, comprising: Obtaining automatic speech recognition features and speaker feature vectors of the target speaker's audio; Input the automatic speech recognition feature and the speaker feature vector into a second sound conversion model, and output the fundamental frequency information of the target speaker audio; The second sound conversion model is trained based on the sound conversion model training method as described in any one of the above items.

[0012] According to the sound conversion method provided by the present invention, after the step of inputting the automatic speech recognition feature and the speaker feature vector into the second sound conversion model and outputting the fundamental frequency information of the target speaker audio, the method further comprises: The fundamental frequency information of the target speaker's audio is subjected to sound conversion processing by a vocoder to obtain audio after sound conversion.

[0013] The present invention also provides a sound conversion model training device, comprising: A first acquisition module is used to acquire multiple speaker audio samples of different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; A first training module, configured to train a preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the audio samples of each speaker, to obtain a first sound conversion model; A second training module is used to train the first sound conversion model based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector to obtain a second sound conversion model; Among them, the speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0014] The present invention also provides a sound conversion device, comprising: A second acquisition module is used to obtain automatic speech recognition features and speaker feature vectors of the target speaker's audio; A conversion module, used for inputting the automatic speech recognition feature and the speaker feature vector into a second sound conversion model, and outputting the fundamental frequency information of the target speaker audio; The second sound conversion model is trained based on the sound conversion model training method as described in any one of the above items.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements any one of the above-mentioned sound conversion model training methods or the sound conversion methods.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described sound conversion model training methods or the above-described sound conversion methods.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned sound conversion model training methods or the above-mentioned sound conversion methods.

[0018] The sound conversion model training method, sound conversion method and device provided by the present invention can construct a diversified training data set by collecting audio samples of multiple different speakers, thereby training a first sound conversion model that can process multiple sound features. Then, the first sound conversion model can be further fine-tuned using the forged audio sample of the target speaker to obtain a second sound conversion model, so that the model can learn the sound features of the target speaker more finely. The obtained model can learn the sound features of the target speaker more finely, improve the similarity of timbre conversion, and effectively reduce timbre leakage. The second sound conversion model is more in line with the sound characteristics of the target speaker, and improves the personalization and similarity of the sound conversion. The second sound conversion model finally trained can output the fundamental frequency information corresponding to the audio of the target speaker based on the input automatic speech recognition features and speaker feature vectors, thereby achieving high-quality timbre conversion and solving the timbre leakage problem caused by training based on multiple audio sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the sound conversion model training method provided by the present invention; Figure 2 A schematic diagram of the forged data generation process provided by the present invention; Figure 3 A schematic diagram of the flow chart of the sound conversion method provided by the present invention; Figure 4 A complete flow chart of sound conversion provided by the present invention; Figure 5 A structural diagram of the sound conversion model provided by the present invention; Figure 6 A schematic diagram of the structure of the sound conversion model training device provided by the present invention; Figure 7 A schematic diagram of the structure of the sound conversion device provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In the related art, traditional voice conversion algorithms are usually trained based on a reconstruction framework, which means that the audio of the target speaker is used during the training process. This practice results in the model's input and output data coming from the same speaker when learning, which does not match the need for feature conversion across speakers in actual voice conversion tasks. Therefore, in practical applications, the voice conversion algorithm may not be able to effectively convert the sound of the source audio into the audio of the target speaker, resulting in the problem of timbre leakage of the source audio.

[0023] Figure 1 is a flow chart of the sound conversion model training method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 110, obtaining multiple speaker audio samples of different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; In the present invention, speaker audio samples of multiple different speakers are obtained to construct a data set containing diverse sound features, and the data set can be used to train a sound conversion model that can understand and imitate the features of different speakers.

[0024] In the present invention, by collecting audio samples from different speakers, the model can learn the common features and individual differences of various sounds, thereby improving its generalization ability.

[0025] In an optional embodiment, audio samples of speakers with a number of 200 speakers and a single-speaking audio duration of more than 1 hour may be obtained. When obtaining the audio samples of speakers, audio samples of various styles may be obtained.

[0026] More specifically, in the present invention, a fundamental frequency label is provided for each audio sample, so that the model can not only learn the timbre and rhythmic characteristics of the sound during the training process, but also accurately capture and convert the pitch information of the sound.

[0027] Step 120, training a preset sound conversion model according to the automatic speech recognition sample features corresponding to the audio samples of each speaker and the speaker sample feature vector to obtain a first sound conversion model; In the present invention, after the speaker audio sample audio is denoised, the continuous audio signal is divided into shorter frames for local analysis. Then, the acoustic features such as the Mel frequency cepstral coefficients of each frame are calculated to capture the automatic speech recognition sample features such as the spectrum information of the speech signal. Techniques such as i-vector or x-vector can also be used to extract the speaker sample feature vector that characterizes the speaker's identity.

[0028] In the present invention, by training a preset sound conversion model, the model learns how to map the input automatic speech recognition features and speaker feature vectors to the target output, that is, the fundamental frequency information of the target speaker. After training, the first sound conversion model obtained can process any input speaker audio sample to ensure that the first sound conversion model has basic reconstruction capabilities, that is, the first sound conversion model in this application has the ability to process audio processing for different speakers, paving the way for subsequent secondary training.

[0029] Step 130, training the first sound conversion model based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector to obtain a second sound conversion model; Among them, the speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0030] In the present invention, the target speaker may specifically refer to any speaker among a plurality of different speakers, and may specifically be a speaker designated by a user.

[0031] In the present invention, based on the real speaker audio sample of the target speaker, a forged speaker audio forged sample is generated by means of fundamental frequency conversion and the like, and a forged automatic speech recognition feature of the forged speaker audio sample is obtained.

[0032] These forged samples retain the characteristics of the target speaker, as well as the fundamental frequency label and speaker feature vector corresponding to the real speaker audio sample. However, the timbre is different from the original audio, providing more diverse training data for the model.

[0033] In the present invention, the first voice conversion model preliminarily trained is fine-tuned through these forged samples to obtain a second voice conversion model. The fine-tuning process enables the model to learn the voice characteristics of the target speaker more finely and improve the timbre similarity after conversion.

[0034] In the present invention, since the second voice conversion model is obtained by retraining on the basis of the first voice conversion model, the second voice conversion model itself has the ability to convert sounds for a variety of different human voices, and after the second training, it can effectively make the data properties of the training and reasoning stages consistent based on the parallel data set of audio with different timbre from the target speaker (but the same content), thereby improving the effect of sound conversion and reducing the problem of timbre leakage.

[0035] In the present invention, the second sound conversion model can output the fundamental frequency information corresponding to the target speaker's audio according to the input automatic speech recognition features and speaker feature vector. It can thus generate a converted speech that is closer to the target speaker's natural voice. This technology can be applied to speech synthesis, video dubbing, speech recognition and other fields, providing higher quality timbre conversion services and enhancing user experience and application practicality.

[0036] In the present invention, by collecting audio samples of multiple different speakers, a diversified training data set can be constructed, thereby training a first sound conversion model that can process multiple sound features. Then, the forged audio samples of the target speaker can be further used for fine-tuning, so that the model can learn the sound features of the target speaker more finely to obtain a second sound conversion model. In the process of training the second sound conversion model, a parallel data set is constructed by generating audio with a different timbre from the target speaker (but the same content), so that the data properties of the training and reasoning stages are consistent, thereby improving the effect of sound conversion and reducing the problem of timbre leakage. The trained second sound conversion model is more in line with the sound characteristics of the target speaker, and the personalization and similarity of the sound conversion are improved. The second sound conversion model finally trained can output the fundamental frequency information corresponding to the audio of the target speaker based on the input automatic speech recognition features and speaker feature vectors, thereby achieving high-quality timbre conversion.

[0037] Optionally, the method for generating a speaker audio forgery sample comprises: Obtaining all audio samples of the target speaker corresponding to the target speaker, and obtaining multiple groups of random number seeds; The pitch and frequency band of each speaker's audio sample are transformed according to the multiple groups of random number seeds to obtain multiple speaker's audio forged samples with adjusted timbre.

[0038] In the present invention, different random number seeds can be pre-designed. The random number seed is a component used to generate random numbers in an algorithm or program, and can be specifically used to generate random pitch amplitude adjustment information and random frequency band change adjustment information.

[0039] Optionally, the pitch conversion and frequency band conversion of each of the speaker audio samples according to the multiple groups of random number seeds to obtain multiple speaker audio forgery samples with adjusted timbre includes: For any group of the random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch-transform the audio samples of each speaker to obtain the audio samples of each speaker after pitch transformation; Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker's audio forged sample with adjusted timbre; Each of the random number seeds is traversed, and the pitch and frequency band of each of the speaker's audio samples are transformed to obtain a plurality of speaker audio forgery samples with adjusted timbre.

[0040] In the present invention, for each set of random number seeds, the pitch of each speaker's audio sample is changed according to the random pitch amplitude adjustment information generated by the seeds, which means changing the fundamental frequency, i.e., the pitch, of the audio sample to simulate the pitch characteristics of different speakers.

[0041] Based on each set of random number seeds, the frequency band of the audio sample is transformed using the random frequency band change adjustment information generated by the seeds. This involves adjusting the spectral envelope of the audio signal to change the timbre and simulate the resonance characteristics of different speakers.

[0042] Frequency band conversion may be achieved by changing the parameters of the filter, such as adjusting the center frequency of the form filter bank to simulate the oral cavity and vocal tract characteristics of different speakers.

[0043] In the present invention, all random number seeds are traversed, the pitch conversion and frequency band conversion processes are repeated, and multiple conversions are performed on each speaker's audio sample. In this way, multiple speaker audio forgery samples with adjusted timbre can be generated, each sample corresponding to a different random number seed, thereby increasing the diversity of samples.

[0044] In the present invention, the timbre of each speaker's audio sample is adjusted through pitch conversion and frequency band conversion to generate a speaker's audio forged sample with adjusted timbre. These forged samples are consistent with the original audio samples in content, but different in timbre, thereby providing diversified training data.

[0045] In an alternative embodiment, Figure 2 The schematic diagram of the forged data generation process provided by the present invention is as follows: Figure 2 As shown, each time data forgery is performed, a set of random number seeds will be designed, and the random number seeds will be used as the base frequency conversion parameters; Extract the fundamental frequency of the target speaker's audio, and determine the amplitude of raising and lowering the pitch based on the set random number seed, thereby preliminarily changing the timbre of the source audio.

[0046] Then, through the low-shelf filter, high-shelf filter and peak filter, the frequency band of the audio is subjected to random amplitude gain or attenuation processing according to random parameters, thereby further changing the timbre of the source audio.

[0047] Optionally, the step of training a preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the speaker audio samples to obtain the first sound conversion model includes: Taking the automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each speaker audio sample as a first training sample, and obtaining a plurality of first training samples; For any first training sample, input the first training sample into the preset sound conversion model, and output fundamental frequency information corresponding to the first training sample; Using a first preset loss function, a first loss value is calculated according to the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein the first preset loss function includes: a reconstruction loss function and an adversarial loss function; When the first loss value is less than a first preset threshold, the training is stopped to obtain a first sound conversion model.

[0048] In the present invention, the automatic speech recognition features corresponding to each speaker audio sample, the speaker sample feature vector and the fundamental frequency label are combined into a first training sample.

[0049] In this way, multiple first training samples are prepared for model training, each of which contains complete input features and target output labels.

[0050] In the present invention, for any first training sample, it is input into a preset sound conversion model. The model outputs corresponding fundamental frequency information according to the input automatic speech recognition features and speaker feature vector. A first preset loss function is used to calculate a loss value according to the fundamental frequency information output by the model and the fundamental frequency label corresponding to the training sample.

[0051] The first preset loss function includes two parts: reconstruction loss function and adversarial loss function.

[0052] Reconstruction loss function: measures the difference between the model output and the true fundamental frequency label to ensure that the model can accurately reconstruct the target fundamental frequency; Adversarial loss function: used in the generative adversarial network to measure the distinction between the generated fundamental frequency information and the true fundamental frequency information, and promote the model to generate more realistic fundamental frequency information.

[0053] When the calculated first loss value is less than the preset threshold, the model is considered to have achieved a certain training effect and training is stopped. This threshold is a pre-set performance indicator used to determine when to stop training to avoid overfitting and ensure that the model has good generalization ability.

[0054] After the training is completed, the obtained first sound conversion model can accurately output fundamental frequency information according to the input automatic speech recognition features and speaker feature vectors for subsequent sound conversion tasks.

[0055] Optionally, the training of the first sound conversion model based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector to obtain the second sound conversion model includes: Taking the forged automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each of the forged audio samples of the speaker as a second training sample, and obtaining a plurality of second training samples; For any second training sample, input the second training sample into the first sound conversion model, and output fundamental frequency information corresponding to the second training sample; Using a second preset loss function, calculating a second loss value according to the fundamental frequency information corresponding to the second training sample and the fundamental frequency label corresponding to the second training sample; wherein the second preset loss function only includes a reconstruction loss function; When the second loss value is less than a second preset threshold, the training is stopped to obtain a second sound conversion model.

[0056] In the present invention, the forged automatic speech recognition features, the speaker sample feature vector and the fundamental frequency label corresponding to each speaker audio forged sample are combined into a second training sample. In this way, multiple second training samples are prepared for fine-tuning the model, each sample including the forged input features and the target output label.

[0057] For any second training sample, it is input into the first sound conversion model. The model outputs the corresponding fundamental frequency information according to the input forged automatic speech recognition sample and the speaker feature vector. A second preset loss function is used to calculate a second loss value according to the fundamental frequency information output by the model and the fundamental frequency label corresponding to the second training sample.

[0058] The second preset loss function only includes the reconstruction loss function, and the training of the model only focuses on minimizing the difference between the output fundamental frequency and the true fundamental frequency label, and does not involve the adversarial loss function. This is because the overall loss may increase significantly in the later stage of model training, which is manifested as obvious errors in the pronunciation of the synthesized audio. After further investigation, it was found that the source of the gradient loss was the learning collapse of the discriminator. Since the sound conversion task is not a reconstruction task, the discriminator is attempted to be removed in the present invention, that is, the adversarial loss is removed while retaining the reconstruction loss. This method enables the training mode based on fundamental frequency forged data to be trained to more steps without gradient explosion, thereby improving the audio similarity without losing the pronunciation of the synthesized audio.

[0059] In the present invention, when the calculated second loss value is less than the preset second threshold value, it is considered that the model has achieved a certain training effect, and the training is stopped at this time. After the training is completed, the obtained second sound conversion model can accurately output the fundamental frequency information according to the input forged automatic speech recognition features and speaker feature vectors, which can be used for subsequent sound conversion tasks, especially in simulating the sound characteristics of the target speaker.

[0060] In the present invention, the training process of the second voice conversion model further optimizes the performance of the model by fine-tuning the first voice conversion model using fake samples, making it better at simulating the voice characteristics of the target speaker. And by using only the reconstruction loss function, the training of the model focuses more on improving the accuracy of voice conversion rather than the fidelity in generative adversarial training, avoiding the gradient explosion problem in the later stage of training.

[0061] Figure 3 A flow chart of the sound conversion method provided by the present invention is shown in FIG. Figure 3 As shown, including: Step 310, obtaining automatic speech recognition features and speaker feature vectors of the target speaker's audio; In the present invention, the automatic speech recognition feature is an acoustic feature extracted from a speech signal, which is used to characterize the properties of the speech signal for speech recognition. The most common automatic speech recognition feature is the Mel-frequency cepstral coefficient. The Mel-frequency cepstral coefficient is obtained by converting the speech signal into a spectrum on the Mel-frequency scale.

[0062] In the present invention, the speaker feature vector is a feature used to characterize the speaker identity information. In the field of speaker recognition, X-vector is a widely used speaker feature vector.

[0063] Step 320, inputting the automatic speech recognition feature and the speaker feature vector into a second sound conversion model, and outputting the fundamental frequency information of the target speaker audio; The second sound conversion model is trained based on the sound conversion model training method as described in any one of the above items.

[0064] In the present invention, these features are used as input data to be fed into the second sound conversion model, which is obtained by retraining the first sound conversion model in the above steps and has learned how to generate the fundamental frequency information of the target speaker based on the input features.

[0065] In the present invention, the second sound conversion model receives the automatic speech recognition features and the speaker feature vector, processes these features and learns the relationship between them and the fundamental frequency of the target speaker through its internal neural network structure. The output of the model is the fundamental frequency information, that is, the representation of the pitch or pitch change. This output is the key in the sound conversion process because it determines the pitch characteristics of the converted audio, thereby affecting the sound similarity in the auditory sense. The fundamental frequency information can effectively adjust the input audio to a timbre similar to that of the target speaker.

[0066] In the present invention, the output fundamental frequency information can be used for further sound synthesis or sound conversion process, such as synthesizing the fundamental frequency information and other acoustic features into an audible audio signal through a vocoder, or used to adjust the pitch of existing audio to match the voice of the target speaker.

[0067] In an alternative embodiment, Figure 4 The complete flow chart of the sound conversion provided by the present invention is as follows: Figure 4 As shown, including: First, prepare multi-person data with different styles; the total number of speakers should be between 100 and 200; and the audio duration of a single speaker should be more than 1 hour.

[0068] Then, the audio data is subjected to automatic speech recognition feature extraction to form the content features of the input audio, and the speaker feature vector is extracted, which is used together with the content features as the input of the decoder; at the same time, the fundamental frequency is extracted as the fundamental frequency label, which is used as the optimization target.

[0069] Then, based on the above data, a multi-person voice conversion model is trained based on the reconstruction framework; For the target speaker, a fundamental frequency forging algorithm is used to synthesize 100-200 sets of parallel data, and the forged automatic speech recognition features of the parallel data are extracted; Based on the pure sound conversion model framework, a single-person sound conversion model is trained on fundamental frequency parallel forged data, and the weights of the multi-person model are loaded to obtain the final sound conversion model.

[0070] In an alternative embodiment, Figure 5 The sound conversion model structure diagram provided by the present invention is as follows: Figure 5 As shown in the figure, first, the process of voice conversion begins by obtaining fake automatic speech recognition features, which are derived from the fundamental frequency forgery source and used as content features. Then, these content features are combined with the speaker sample feature vector of the fundamental frequency forgery source to form the input of the model. These input data are fed into the voice conversion model, and the model then outputs the synthesized fundamental frequency information.

[0071] The synthesized fundamental frequency information is used together with the corresponding fundamental frequency label of the fundamental frequency forgery source to calculate the loss value. The calculation of the loss value can include two main parts: reconstruction loss and adversarial loss. The reconstruction loss ensures the consistency of the fundamental frequency information output by the model with the target fundamental frequency label, while the adversarial loss is used to improve the quality and fidelity of the synthesized fundamental frequency.

[0072] Finally, the synthesized fundamental frequency information is fed into the vocoder, which is responsible for converting this fundamental frequency information into audible synthesized audio. In this way, through the structural design of the model and accurate loss calculation, high-quality audio synthesis can be achieved, generating audio output that matches the timbre of the target speaker.

[0073] The device provided by the present invention is described below. The device described below and the method described above can be referenced to each other.

[0074] Figure 6 A schematic diagram of the structure of the sound conversion model training device provided by the present invention is shown in FIG. Figure 6 As shown, including: The first acquisition module 610 is used to acquire multiple speaker audio samples of different speakers; wherein each speaker audio sample corresponds to a fundamental frequency label; The first training module 620 is used to train the preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the audio samples of each speaker, so as to obtain a first sound conversion model; The second training module 630 is used to train the first sound conversion model based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector to obtain a second sound conversion model; Among them, the speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

[0075] Optionally, the device is also used for: Obtaining all audio samples of the target speaker corresponding to the target speaker, and obtaining multiple groups of random number seeds; The pitch and frequency band of each speaker's audio sample are transformed according to the multiple groups of random number seeds to obtain multiple speaker's audio forged samples with adjusted timbre.

[0076] Optionally, the device is also used for: For any group of the random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch-transform the audio samples of each speaker to obtain the audio samples of each speaker after pitch transformation; Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker's audio forged sample with adjusted timbre; Each of the random number seeds is traversed, and the pitch and frequency band of each of the speaker's audio samples are transformed to obtain a plurality of speaker audio forgery samples with adjusted timbre.

[0077] Optionally, the device is also used for: Taking the automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each speaker audio sample as a first training sample, and obtaining a plurality of first training samples; For any first training sample, input the first training sample into the preset sound conversion model, and output fundamental frequency information corresponding to the first training sample; Using a first preset loss function, a first loss value is calculated according to the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein the first preset loss function includes: a reconstruction loss function and an adversarial loss function; When the first loss value is less than a first preset threshold, the training is stopped to obtain a first sound conversion model.

[0078] Optionally, the device is also used for: Taking the forged automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each of the forged audio samples of the speaker as a second training sample, and obtaining a plurality of second training samples; For any second training sample, input the second training sample into the first sound conversion model, and output fundamental frequency information corresponding to the second training sample; Using a second preset loss function, calculating a second loss value according to the fundamental frequency information corresponding to the second training sample and the fundamental frequency label corresponding to the second training sample; wherein the second preset loss function only includes a reconstruction loss function; When the second loss value is less than a second preset threshold, the training is stopped to obtain a second sound conversion model.

[0079] Figure 7 A schematic diagram of the structure of the sound conversion device provided by the present invention, as shown in Figure 7 As shown, including: The second acquisition module 710 is used to obtain the automatic speech recognition features and speaker feature vector of the target speaker audio; The conversion module 720 is used to input the automatic speech recognition feature and the speaker feature vector into a second sound conversion model, and output the fundamental frequency information of the target speaker audio; The second sound conversion model is trained based on the sound conversion model training method as described in any one of the above items.

[0080] Optionally, the device is also used for: The fundamental frequency information of the target speaker's audio is subjected to sound conversion processing by a vocoder to obtain audio after sound conversion.

[0081] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute a sound conversion model training method or a sound conversion method, which includes: obtaining speaker audio samples of multiple different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; training a preset sound conversion model according to the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the speaker audio samples to obtain a first sound conversion model; training the first sound conversion model based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each forged speaker audio sample of the target speaker to obtain a second sound conversion model; wherein the forged speaker audio sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vectors corresponding to the speaker audio.

[0082] Or, obtain automatic speech recognition features and speaker feature vectors of the target speaker's audio; input the automatic speech recognition features and speaker feature vectors into a second sound conversion model, and output the fundamental frequency information of the target speaker's audio; wherein the second sound conversion model is trained based on the sound conversion model training method as described in any of the above items.

[0083] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0084] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the sound conversion model training method or the sound conversion method provided by the above-mentioned methods, and the method includes: obtaining speaker audio samples of multiple different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; according to the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the speaker audio samples, a preset sound conversion model is trained to obtain a first sound conversion model; based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each speaker audio forged sample of the target speaker, the first sound conversion model is trained to obtain a second sound conversion model; wherein the speaker audio forged sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vectors corresponding to the speaker audio.

[0085] Or, obtain automatic speech recognition features and speaker feature vectors of the target speaker's audio; input the automatic speech recognition features and speaker feature vectors into a second sound conversion model, and output the fundamental frequency information of the target speaker's audio; wherein the second sound conversion model is trained based on the sound conversion model training method as described in any of the above items.

[0086] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the sound conversion model training method or the sound conversion method provided by the above-mentioned methods, the method comprising: obtaining speaker audio samples of multiple different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; training a preset sound conversion model according to the automatic speech recognition sample features and speaker sample feature vectors corresponding to each of the speaker audio samples to obtain a first sound conversion model; training the first sound conversion model based on the forged automatic speech recognition features and speaker sample feature vectors corresponding to each forged speaker audio sample of the target speaker to obtain a second sound conversion model; wherein the forged speaker audio sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vectors corresponding to the speaker audio.

[0087] Or, obtain automatic speech recognition features and speaker feature vectors of the target speaker's audio; input the automatic speech recognition features and speaker feature vectors into a second sound conversion model, and output the fundamental frequency information of the target speaker's audio; wherein the second sound conversion model is trained based on the sound conversion model training method as described in any of the above items.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a sound conversion model, characterized in that: include: Acquire multiple speaker audio samples of different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; Training a preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the speaker audio samples to obtain a first sound conversion model; Based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector, the first sound conversion model is trained to obtain a second sound conversion model; Among them, the speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

2. The method for training a voice conversion model according to claim 1, characterized in that: The method for generating a speaker audio forgery sample comprises: Obtaining all audio samples of the target speaker corresponding to the target speaker, and obtaining multiple groups of random number seeds; The pitch and frequency band of each speaker's audio sample are transformed according to the multiple groups of random number seeds to obtain multiple speaker's audio forged samples with adjusted timbre.

3. The method for training a voice conversion model according to claim 2, characterized in that: The pitch conversion and frequency band conversion of each of the speaker audio samples according to the multiple groups of random number seeds to obtain multiple speaker audio forgery samples with adjusted timbre includes: For any group of the random number seeds, based on the random pitch amplitude adjustment information corresponding to the random number seeds, pitch-transform the audio samples of each speaker to obtain the audio samples of each speaker after pitch transformation; Based on the random frequency band change adjustment information corresponding to the random number seed, the frequency band of each speaker's audio sample is transformed to obtain a speaker's audio forged sample with adjusted timbre; Each of the random number seeds is traversed, and the pitch and frequency band of each of the speaker's audio samples are transformed to obtain a plurality of speaker audio forgery samples with adjusted timbre.

4. The method for training a voice conversion model according to claim 1, characterized in that: The method of training a preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the speaker audio samples to obtain a first sound conversion model includes: Taking the automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each speaker audio sample as a first training sample, and obtaining a plurality of first training samples; For any first training sample, input the first training sample into the preset sound conversion model, and output fundamental frequency information corresponding to the first training sample; Using a first preset loss function, a first loss value is calculated according to the fundamental frequency information and the fundamental frequency label corresponding to the first training sample; wherein the first preset loss function includes: a reconstruction loss function and an adversarial loss function; When the first loss value is less than a first preset threshold, the training is stopped to obtain a first sound conversion model.

5. The method for training a voice conversion model according to claim 1, characterized in that: The method of training the first sound conversion model based on the forged automatic speech recognition features corresponding to the forged audio samples of each speaker of the target speaker and the speaker sample feature vector to obtain the second sound conversion model includes: Taking the forged automatic speech recognition sample features, the speaker sample feature vector, and the fundamental frequency label corresponding to each of the forged audio samples of the speaker as a second training sample, and obtaining a plurality of second training samples; For any second training sample, input the second training sample into the first sound conversion model, and output fundamental frequency information corresponding to the second training sample; Using a second preset loss function, calculating a second loss value according to the fundamental frequency information corresponding to the second training sample and the fundamental frequency label corresponding to the second training sample; wherein the second preset loss function only includes a reconstruction loss function; When the second loss value is less than a second preset threshold, the training is stopped to obtain a second sound conversion model.

6. A sound conversion method, characterized in that: include: Obtaining automatic speech recognition features and speaker feature vectors of the target speaker's audio; Input the automatic speech recognition feature and the speaker feature vector into a second sound conversion model, and output fundamental frequency information corresponding to the target speaker audio; The second sound conversion model is trained based on the sound conversion model training method according to any one of claims 1 to 5.

7. The sound conversion method according to claim 6, characterized in that: After the step of inputting the automatic speech recognition feature and the speaker feature vector into a second sound conversion model and outputting the fundamental frequency information of the target speaker audio, the method further comprises: The fundamental frequency information of the target speaker's audio is subjected to sound conversion processing by a vocoder to obtain audio after sound conversion.

8. A sound conversion model training device, characterized in that: include: A first acquisition module is used to acquire multiple speaker audio samples of different speakers; wherein each of the speaker audio samples corresponds to a fundamental frequency label; A first training module, configured to train a preset sound conversion model according to the automatic speech recognition sample features and the speaker sample feature vectors corresponding to the audio samples of each speaker, to obtain a first sound conversion model; A second training module is used to train the first sound conversion model based on the forged automatic speech recognition features corresponding to each forged audio sample of the target speaker and the speaker sample feature vector to obtain a second sound conversion model; Among them, the speaker audio forgery sample is obtained based on the real speaker audio sample of the target speaker; the second sound conversion model is used to output the fundamental frequency information corresponding to the speaker audio based on the automatic speech recognition features and speaker feature vector corresponding to the speaker audio.

9. A sound conversion device, characterized in that: include: A second acquisition module is used to obtain automatic speech recognition features and speaker feature vectors of the target speaker's audio; A conversion module, used for inputting the automatic speech recognition feature and the speaker feature vector into a second sound conversion model, and outputting the fundamental frequency information of the target speaker audio; The second sound conversion model is trained based on the sound conversion model training method according to any one of claims 1 to 5.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the sound conversion model training method as described in any one of claims 1 to 5 or the sound conversion method as described in any one of claims 6-7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the sound conversion model training method as described in any one of claims 1 to 5 or the sound conversion method as described in any one of claims 6-7.

Images