Vocoder training method, singing sound conversion method, electronic equipment and storage medium

Abstract

The invention provides a vocoder training method, a singing sound conversion method, electronic equipment and a storage medium, and relates to the technical field of audio processing. The vocoder training method comprises the steps that sample standard human voice and sample conversion human voice corresponding to a sample song are acquired, the sample standard human voice and the sample conversion human voice are singing audios of the sample song with the same timbre, the sample conversion human voice is singing audios of the sample song with artifacts and high pitch drift, and the sample standard human voice and the sample conversion human voice are acquired; the sample standard human voice is the standard singing audio of the sample song; pitch extraction is carried out on the sample standard human voice to obtain sample standard pitch features; and training the initial vocoder according to the sample standard human voice, the sample conversion human voice and the sample standard pitch feature to obtain the vocoder. In the training process, the initial vocoder is guided to remove artifacts and correct tones, so that the trained vocoder can realize the functions of removing artifacts and pitch drift, and the quality of converted songs can be improved by utilizing the vocoder.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and more specifically, to a vocoder training method, a singing voice conversion method, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence technology, the use of artificial intelligence in audio processing has received widespread attention, among which the timbre conversion of human voices in songs has become a research hotspot.

[0003] In related technologies, users upload reference timbre and target song, convert the vocal timbre in the target song into the reference timbre, and obtain the converted song.

[0004] However, in the related technologies, artifacts such as "bubble sound", "mechanical sound", "gag" and pitch drift appear in the converted songs. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a vocoder training method, a singing voice conversion method, an electronic device, and a storage medium, so as to solve the aforementioned technical problems existing in the related technologies.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a vocoder training method, the method comprising: Obtain the standard vocal sample and the converted vocal sample corresponding to the sample song. The standard vocal sample and the converted vocal sample are the singing audio of the sample song with the same timbre. The converted vocal sample is the singing audio of the sample song with artifacts and high-pitched drift. The standard vocal sample is the standard singing audio of the sample song. The pitch of the sample standard human voice is extracted to obtain the sample standard pitch features; The initial vocoder is trained based on the sample standard human voice, the sample converted human voice, and the sample standard pitch features to obtain the vocoder.

[0007] Optionally, obtaining the standard human voice and converted human voice corresponding to the sample song includes: The vocals and accompaniment in the sample song are separated to obtain the separated vocal samples. The sample human voice is separated and processed by de-reverberation and chordation to obtain the sample standard human voice; Based on the timbre configuration of the sample standard human voice, the sample standard human voice is converted to obtain the sample converted human voice.

[0008] Optionally, the step of extracting pitch from the sample standard human voice to obtain sample standard pitch features includes: Extract the pitch curve of the standard human voice sample frame by frame; The sample pitch curves are transformed to obtain the sample pitch features; The sample pitch features are normalized to obtain normalized sample pitch features; The normalized sample pitch features are smoothed and the note boundaries are aligned to obtain the sample standard pitch features.

[0009] Optionally, training the initial vocoder based on the sample standard human voice, the sample converted human voice, and the sample standard pitch features to obtain the vocoder includes: The fusion module in the initial vocoder is used to fuse the sample converted human voice and the sample standard pitch features to obtain sample fusion features. The processing module in the initial vocoder is used to process the sample fusion features output by the fusion module to obtain the sample predicted human voice; Calculate multiple loss function values ​​based on the predicted human voice from the sample and the standard human voice from the sample; Based on the multiple loss function values, the model parameters of the initial vocoder are updated to obtain the vocoder.

[0010] Optionally, the fusion module in the initial vocoder is used to fuse the sample-converted human voice and the sample standard pitch features to obtain sample fusion features, including: The channel splicing module in the fusion module is used to splice the converted human voice of the sample and the standard pitch feature of the sample to obtain the spliced ​​sample features; Perform an affine transformation on the spliced ​​sample features to obtain the transformed sample features; The gated residual module in the fusion module is used to fuse the spliced ​​sample features output by the channel splicing module and the sample converted human voice to obtain the sample fusion features.

[0011] Optionally, the step of calculating multiple loss function values ​​based on the predicted human voice from the sample and the standard human voice from the sample includes: Based on the predicted human voice from the sample and the standard human voice from the sample, calculate the multi-scale short-time Fourier transform reconstruction loss, adversarial loss, perceptual loss, and pitch consistency loss, respectively.

[0012] Secondly, embodiments of this application also provide a singing voice conversion method, the method comprising: The vocals and accompaniment in the original song are separated to obtain the accompaniment and the original vocal timbre; Based on the original vocal timbre, a timbre conversion is performed to obtain the target vocal timbre. The target timbre human voice is encoded using a vocoder to obtain the target timbre standard human voice, wherein the vocoder is a vocoder obtained by the vocoder training method described in any of the first aspects above. The target song is generated based on the target timbre standard vocal and the accompaniment.

[0013] Optionally, the step of converting the original timbre of the human voice to obtain the target timbre includes: The original vocal timbre is subjected to de-reverb and de-chord processing to obtain the processed original vocal timbre. The processed original vocal timbre is converted to obtain the target vocal timbre.

[0014] Thirdly, embodiments of this application also provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the method described in any of the first aspects above.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when read and executed, implements the method described in any of the first aspects above.

[0016] Fifthly, embodiments of this application also provide a vocoder training apparatus, comprising: The acquisition module is used to acquire the standard vocal sample and the converted vocal sample corresponding to the sample song. The standard vocal sample and the converted vocal sample are the singing audio of the sample song with the same timbre. The converted vocal sample is the singing audio of the sample song with artifacts and high-pitched drift. The standard vocal sample is the standard singing audio of the sample song. The extraction module is used to extract the pitch of the sample standard human voice to obtain the sample standard pitch features; The training module is used to train the initial vocoder based on the sample standard human voice, the sample converted human voice, and the sample standard pitch features to obtain the vocoder.

[0017] Optionally, the acquisition module is specifically used to separate the vocals and accompaniment in the sample song to obtain the sample separated vocals; to perform de-reverb and harmonization processing on the sample separated vocals to obtain the sample standard vocals; and to convert the sample standard vocals according to the timbre configuration of the sample standard vocals to obtain the sample converted vocals.

[0018] Optionally, the extraction module is specifically used to extract sample pitch curves frame by frame from the sample standard human voice; perform formal transformation on the sample pitch curves to obtain sample pitch features; perform normalization processing on the sample pitch features to obtain normalized sample pitch features; and perform smoothing processing and note boundary alignment processing on the normalized sample pitch features to obtain the sample standard pitch features.

[0019] Optionally, the training module is specifically used to use the fusion module in the initial vocoder to fuse the sample converted human voice and the sample standard pitch features to obtain sample fusion features; use the processing module in the initial vocoder to process the sample fusion features output by the fusion module to obtain sample predicted human voice; calculate multiple loss function values ​​based on the sample predicted human voice and the sample standard human voice; and update the model parameters of the initial vocoder based on the multiple loss function values ​​to obtain the vocoder.

[0020] Optionally, the training module is specifically used to use the channel splicing module in the fusion module to splice the sample-converted human voice and the sample standard pitch features to obtain spliced ​​sample features; to perform an affine transformation on the spliced ​​sample features to obtain transformed sample features; and to use the gated residual module in the fusion module to fuse the spliced ​​sample features output by the channel splicing module and the sample-converted human voice to obtain the sample fusion features.

[0021] Optionally, the training module is specifically used to calculate the multi-scale short-time Fourier transform reconstruction loss value, adversarial loss value, perceptual loss value, and pitch consistency loss value based on the human voice predicted from the sample and the standard human voice from the sample, respectively.

[0022] Sixthly, embodiments of this application also provide a singing voice conversion device, including: The separation module is used to separate the vocals and accompaniment in the original song to obtain the accompaniment and the original vocal timbre. The conversion module is used to convert the original timbre of the human voice to obtain the target timbre of the human voice. The processing module is used to perform sound encoding processing on the target timbre human voice using a vocoder to obtain the target timbre standard human voice, wherein the vocoder is a vocoder obtained by using the vocoder training method described in any of the first aspects above. The generation module is used to generate a target song based on the target timbre standard vocals and the accompaniment.

[0023] Optionally, the conversion module is specifically used to perform de-reverb and de-chord processing on the original timbre vocal to obtain the processed original timbre vocal; and to perform timbre conversion on the processed original timbre vocal to obtain the target timbre vocal.

[0024] The beneficial effects of this application are as follows: This application provides a vocoder training method, which may include: acquiring sample standard vocals and sample converted vocals corresponding to sample songs, wherein the sample standard vocals and sample converted vocals are singing audios of sample songs with the same timbre, the sample converted vocals are singing audios of sample songs with artifacts and high-pitched drift, and the sample standard vocals are the standard singing audios of sample songs; extracting pitch from the sample standard vocals to obtain sample standard pitch features; and training an initial vocoder based on the sample standard vocals, sample converted vocals, and sample standard pitch features to obtain a vocoder. The sample converted vocals are samples with artifacts and high-pitched drift, the sample standard vocals are the supervised target, and the sample standard pitch features are the conditional prior. During training, the initial vocoder is guided to remove artifacts and correct pitch. The vocoder trained in this way can achieve the functions of removing artifacts and correcting pitch drift, thereby improving the quality of the converted songs. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 2 ; Figure 3 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 3 ; Figure 4A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 4 ; Figure 5 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 5 ; Figure 6 A flowchart illustrating a singing voice conversion method provided in this application embodiment. Figure 1 ; Figure 7 A flowchart illustrating a singing voice conversion method provided in this application embodiment. Figure 2 ; Figure 8 This is a schematic diagram of the structure of a vocoder training device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a singing voice conversion device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0028] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0030] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0032] This application provides a vocoder training method and a song conversion method, which are applied to electronic devices. The electronic devices can be terminal devices, which can be any of the following: computer devices, laptops, tablets, and smartphones.

[0033] The following explains a vocoder training method provided in the embodiments of this application.

[0034] Figure 1 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method may include: S101. Obtain the standard human voice of the sample song and the converted human voice of the sample song.

[0035] Among them, the standard sample vocal and the converted sample vocal are the singing audio of sample songs with the same timbre, the converted sample vocal is the singing audio of sample songs with artifacts and high-pitched drift, and the standard sample vocal is the standard singing audio of sample songs.

[0036] In some implementations, a standard human voice is determined based on a sample song, and then converted to obtain a converted human voice. The standard human voice and the converted human voice exist in pairs, with one standard human voice and one converted human voice corresponding to one sample song, which are used to train the generation mapping for artifact removal and audio correction in the initial vocoder.

[0037] In this embodiment, the sample-converted human voice is the singing audio of a sample song with artifacts such as "bubble sound," "mechanical sound," and "gags," as well as high-pitched drift and unstable pitch. The sample standard human voice is the standard singing audio of a sample song that has no or only a very small amount of artifacts, high-pitched drift, and unstable pitch. The singing audio specifically refers to the melody sung by a person, without accompaniment. Both the sample-converted human voice and the sample standard human voice are the sound of a person singing.

[0038] It should be noted that the number of sample songs is multiple, which may be greater than or equal to tens of thousands of songs. These sample songs are from multiple genres and languages, covering a wide range of rhythms or modes.

[0039] S102. Extract the pitch of the standard human voice sample to obtain the standard pitch features of the sample.

[0040] Among them, the standard pitch feature of the sample refers to the normalized pitch feature.

[0041] In some implementations, a preset pitch extraction algorithm is used to extract the pitch of the sample standard human voice to obtain the sample standard pitch features. For example, the preset pitch extraction algorithm may be: CREPE (Convolutional Representation for pitch Estimation, a pitch prediction algorithm based on convolutional neural networks), Harvest (Harmonic Regression for pitch Estimation, pitch estimation based on harmonic regression), or the DIO algorithm (a pitch extraction algorithm based on Mel frequency cepstral analysis).

[0042] It is worth noting that the standard pitch features of the samples are unrelated to timbre. The standard pitch features of the samples are used as a conditional prior during the training process of the vocoder to guide the vocoder in artifact removal and pitch correction.

[0043] S103. Based on the standard human voice sample, the converted human voice sample, and the standard pitch characteristics of the sample, train the initial vocoder to obtain the vocoder.

[0044] In one possible implementation, the model parameters of the initial vocoder are iteratively updated based on the sample standard human voice, the sample converted human voice, and the sample standard pitch features. This process learns a joint mapping for artifact removal and pitch correction until a preset termination training condition is met, resulting in the vocoder. During training, the sample standard pitch features are used as the training target, and the sample standard human voice is used as the reference feature.

[0045] In addition, during the training process, DTW (Dynamic Time Warping, an algorithm for aligning time series data of different lengths) or acoustic events are used to pair and supervise the standard human voice samples and the transformed human voice samples after they are aligned on the time axis. This can alleviate the adverse effects of time drift on training and learning.

[0046] In summary, this application provides a vocoder training method, which may include: acquiring sample standard vocals and sample converted vocals corresponding to sample songs, wherein the sample standard vocals and sample converted vocals are singing audios of sample songs with the same timbre, the sample converted vocals are singing audios of sample songs with artifacts and pitch shift, and the sample standard vocals are the standard singing audios of sample songs; extracting pitch from the sample standard vocals to obtain sample standard pitch features; and training an initial vocoder based on the sample standard vocals, sample converted vocals, and sample standard pitch features to obtain a vocoder. The sample converted vocals are samples with artifacts and pitch shift, the sample standard vocals are the supervised target, and the sample standard pitch features are the conditional prior. During training, the initial vocoder is guided to remove artifacts and correct pitch. The vocoder trained in this way can achieve the functions of artifact removal and pitch shift correction, thereby improving the quality of the converted songs.

[0047] Optionally, Figure 2 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, the process of obtaining the standard human voice corresponding to the sample song and converting the sample into human voice in S101 above may include: S201. Separate the vocals and accompaniment in the sample song to obtain the separated vocals.

[0048] It should be noted that vocals and accompaniment can be processed simultaneously for multiple sample songs to obtain multiple samples of separated vocals, or vocals and accompaniment can be processed sequentially for multiple sample songs to obtain multiple samples of separated vocals. This application embodiment does not impose specific limitations on this.

[0049] S202. Perform de-reverberation and chord processing on the sample separated human voice to obtain the sample standard human voice.

[0050] In this process, the sample vocals are separated and de-reverberated and chordalized, thus preserving only the cleanest possible solo vocal parts of the sample standard vocals.

[0051] Similarly, multiple sample human voices can be sequentially processed by dereverberation and chordation to obtain multiple sample standard human voices; or multiple sample human voices can be processed by dereverberation and chordation simultaneously to obtain multiple sample standard human voices. This application does not impose specific limitations on this.

[0052] S203. Based on the timbre configuration of the sample standard human voice, convert the sample standard human voice to obtain the sample converted human voice.

[0053] In some implementations, a sample standard human voice is input, and the sample standard human voice's own timbre is used as a reference timbre conversion configuration. The sample standard human voice is converted according to the reference timbre, that is, a "self-conversion" is performed. The timbre is kept consistent before and after the conversion, and typical artifacts and pitch drift in the conversion chain are introduced to obtain the sample converted human voice.

[0054] It is important to note that the input statistical characteristics of the sample-converted human voice are close to those of the real online conversion of the incompletely separated human voice timbre in the target song into the reference timbre, which can significantly reduce the gap between training and inference distributions.

[0055] In addition, the sample-converted human voice is used as the hard sample, and the sample standard human voice is used as the supervision target. The sample-converted human voice and the sample standard human voice form a one-to-one data pair. After frame-level alignment, the sample-converted human voice and the sample standard human voice are used to train the initial vocoder.

[0056] Optionally, Figure 3 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 3 ,like Figure 3 As shown, the process of extracting the pitch of the sample standard human voice in S102 above to obtain the sample standard pitch features may include: S301. Extract the pitch curve of the standard human voice sample frame by frame.

[0057] In some implementations, the CREPE algorithm, Harvest algorithm, or DIO algorithm is used to extract sample pitch curves frame by frame from the sample standard human voice. The sample pitch curve can be represented as F0.

[0058] S302. Perform formal transformation on the sample pitch curve to obtain the sample pitch characteristics.

[0059] The sample pitch curves are represented using log-F0 or semitone scales that are robust to pitch translation, thus obtaining the sample pitch features.

[0060] S303. Normalize the sample pitch features to obtain normalized sample pitch features.

[0061] The purpose of normalizing the pitch features of the samples is to eliminate differences in timbre, singer-related information, and mode.

[0062] In one possible implementation, the sample pitch features can be normalized using segmented or sliding window z-scores (standard scores). Alternatively, short phrase-level mean and variance can be used for normalization. Another approach is a combination of relative pitch contour and global curve smoothing to obtain normalized sample pitch features. Here, relative pitch contour refers to ΔF0 / relative semitone shift. The normalization method can be selected based on data statistics or used in combination. Injecting a 10-20% normalization perturbation during vocoder training improves robustness.

[0063] It should be noted that the sliding window z-score normalizes log-F0 within approximately 1 second of the sliding window, and the window adaptively expands to the phrase boundary; silent segments are processed using masking. This scheme remains stable during modal changes and modulations, and can express relative pitch contours. The combined strategy of semitone shift and curve smoothing uses the phrase mean as a reference to generate a relative semitone shift curve, and suppresses abrupt changes through bidirectional filtering and first-order difference regularization; light alignment is performed at note boundaries.

[0064] In addition, the normalized sample pitch features only express relative fluctuations and local pitch accuracy, without carrying the singer's identity or absolute tonality.

[0065] S304. The normalized sample pitch features are smoothed and the note boundaries are aligned to obtain the standard pitch features of the samples.

[0066] In some implementations, Savitzky-Golay (a sliding window algorithm based on least squares polynomial fitting) or bidirectional mean filtering is used to smooth the normalized sample pitch features, and an optional pitch alignment network is used to align the note boundaries to obtain the sample standard pitch features.

[0067] Optionally, Figure 4 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 4 ,like Figure 4 As shown, the process of training the initial vocoder based on the sample standard human voice, the sample converted human voice, and the sample standard pitch features in S103 above, to obtain the vocoder, may include: S401. Using the fusion module in the initial vocoder, the sample converted human voice and the standard pitch features of the sample are fused to obtain the sample fusion features.

[0068] Among these features, the sample-converted human voice can be used for Mel-spectrum feature or decoder feature hiding representation, reflecting the input artifacts and the underlying pronunciation. The sample standard pitch features provide explicit pitch priors to guide pitch correction.

[0069] In some implementations, the initial vocoder is used to access two types of conditions: sample-converted human voice and sample standard pitch features. Then, the fusion module in the initial vocoder is used to fuse the sample-converted human voice and sample standard pitch features to obtain sample fusion features, so as to avoid excessive constraints on timbre and articulation timing by pitch prior.

[0070] S402. Using the processing module in the initial vocoder, the sample fusion features output by the fusion module are processed to obtain the sample predicted human voice.

[0071] The discriminator in the initial vocoder uses a combination of multi-scale and multi-cycle methods.

[0072] S403. Calculate multiple loss function values ​​based on the predicted human voice from the sample and the standard human voice from the sample.

[0073] S404. Based on multiple loss function values, update the model parameters of the initial vocoder to obtain the vocoder.

[0074] Optionally, the vocoder can be BigVGAN (a general neural vocoder based on generative adversarial networks). The conditional branch is modified based on BigVGAN, fusing sample converted human voice and sample standard pitch features, with the sample standard human voice as the supervised target, and multiple joint loss functions are used for training.

[0075] In this embodiment, the initial vocoder training configuration includes the Adam optimizer (Adaptive Moment Estimation, an optimization algorithm combining gradient descent and momentum), piecewise gain warmup (a method for optimizing the learning rate), a weighted discriminator, mixed precision and gradient accumulation, and data augmentation with noise or reverberation lightweight injection to improve robustness.

[0076] Optionally, Figure 5 A flowchart illustrating a vocoder training method provided in this application embodiment. Figure 5 ,like Figure 5 As shown, the process in S401 above, which uses the fusion module in the initial vocoder to fuse the sample-converted human voice and the standard pitch features of the sample to obtain the sample fusion features, may include: S501. Using the channel splicing module in the fusion module, the sample converted human voice and the standard pitch features of the sample are spliced ​​to obtain the spliced ​​sample features.

[0077] In some implementations, the standard pitch features of the sample are transmitted to the channel splicing module through the conditional feature path, and the converted human voice of the sample is transmitted to the channel splicing module through the main feature path. The channel splicing module is used to splice the converted human voice of the sample and the standard pitch features of the sample to obtain the spliced ​​sample features.

[0078] S502. Perform an affine transformation on the spliced ​​sample features to obtain the transformed sample features.

[0079] S503. Using the gated residual module in the fusion module, the spliced ​​sample features output by the channel splicing module and the sample converted human voice are fused to obtain the sample fusion features.

[0080] In this embodiment, a channel splicing module and a gated residual module can be used to fuse the sample-converted human voice and the sample standard pitch features to obtain sample fusion features; alternatively, a light attention method can be used to fuse the sample-converted human voice and the sample standard pitch features to obtain sample fusion features.

[0081] Optionally, the process of calculating multiple loss function values ​​based on the sample predicted human voice and the sample standard human voice in S403 above may include: Based on the predicted human voice from the sample and the standard human voice from the sample, the multi-scale short-time Fourier transform reconstruction loss, adversarial loss, perceptual loss, and pitch consistency loss are calculated respectively.

[0082] Among them, joint loss training is adopted, including adversarial, reconstruction, perception and pitch consistency loss.

[0083] It should be noted that the multi-scale short-time Fourier transform reconstruction loss value ensures spectral domain consistency and harmonic structure integrity, the adversarial loss value suppresses metallicity and digital artifacts, the perceptual loss value can improve naturalness and intelligibility, and the pitch consistency loss value aligns the estimated F0 after decoding with the normalized pitch denormalization or relative contour to suppress out-of-tune pitch.

[0084] In addition, the vocoder trained in the embodiments of this application is significantly better than the baseline in terms of "bubble detection rate", "subjective pitch score" and "sentence-level naturalness MOS", and is more robust to extreme reverberation and mild separation crosstalk.

[0085] Optionally, Figure 6 A flowchart illustrating a singing voice conversion method provided in this application embodiment. Figure 1 ,like Figure 6 As shown, the method may include: S601: Separate the vocals and accompaniment from the original song to obtain the accompaniment and the original vocal timbre.

[0086] Among them, the original timbre of the human voice refers to the original timbre of the original song, that is, the tune sung by a person using the original timbre, without accompaniment.

[0087] S602. Perform timbre conversion based on the original timbre of the human voice to obtain the target timbre of the human voice.

[0088] The target vocal tone refers to the audio recording sung with the target vocal tone of the original song, that is, the melody sung by a person using the target vocal tone. It does not include accompaniment.

[0089] It should be noted that both the original and target vocal tones are sung to the tunes of the original song, the difference being that their timbres are different, and the target vocal tones contain artifacts and pitch shift.

[0090] S603. Use a vocoder to encode the target timbre human voice to obtain the target timbre standard human voice.

[0091] Among them, the vocoder is a vocoder obtained by using the above-mentioned vocoder training method, which has the ability to remove artifacts and correct speech.

[0092] In some implementations, a vocoder is used to extract features and normalize pitch from the target timbre human voice, and artifact removal and pitch correction are performed on the target timbre human voice to obtain the target timbre standard human voice, which is a high-fidelity audio.

[0093] S604. Generate the target song based on the target timbre standard vocals and accompaniment.

[0094] In this embodiment, the target vocal timbre standard vocal and accompaniment are mixed to generate the target song. Compared to the original song, the accompaniment and the melody sung by the singer remain the same, but the vocal timbre has changed, thus achieving a cover song using the target vocal timbre.

[0095] In summary, this application provides a method for voice conversion, comprising: separating the vocals and accompaniment in an original song to obtain the accompaniment and the original vocal timbre; performing timbre conversion on the original vocal timbre to obtain a target vocal timbre; using a vocoder to encode the target vocal timbre to obtain a target standard vocal timbre, wherein the vocoder is a vocoder obtained using the aforementioned vocoder training method; and generating a target song based on the target standard vocal timbre and the accompaniment. Using a vocoder to encode the target vocal timbre significantly reduces artifacts and pitch drift in the generated target song, improving the quality and stability of the generated target song.

[0096] Optionally, Figure 7 A flowchart illustrating a singing voice conversion method provided in this application embodiment. Figure 2 ,like Figure 7 As shown, the process of converting the original timbre of the human voice to obtain the target timbre in S602 above may include: S701. Perform de-reverb and de-chord processing on the original vocal timbre to obtain the processed original vocal timbre.

[0097] In some implementations, WPE (weighted prediction error, an algorithm for removing reverberation from speech signals) or frequency domain techniques are used to deverbear the original timbre of the human voice; synthesis suppression and extraction of the main melody of the human voice are then performed to obtain the processed original timbre of the human voice.

[0098] S702. Perform timbre conversion on the processed original vocal timbre to obtain the target vocal timbre.

[0099] In practical applications, A / B testing on actual users showed that complaints about bubbly vocals and off-key singing decreased by 30% to 50%, and the retention and sharing rates of cover songs significantly improved.

[0100] In summary, the vocoder training method and singing voice conversion method provided in this application, through "self-conversion"—that is, sample-converted human voices simulating real-world online difficulty distributions—and using sample standard pitch features to provide robust pitch correction priors, directly learn the joint mapping of "artifact removal + pitch correction" during vocoder training. This overcomes the bottleneck of relying solely on separation and upstream modeling, and possesses the advantages of simple implementation, strong generalization, and online deployment. Specifically, constructing difficult samples using sample-converted human voices and pairing them with sample standard human voices for supervision significantly reduces the training-inference distribution gap. Using timbre-independent sample standard pitch features as conditional priors guides the vocoder to simultaneously remove artifacts and correct pitch. This significantly reduces artifacts such as bubbly sounds and glitches, as well as pitch shifts, with low online integration costs and a significantly improved overall listening experience.

[0101] The following describes the vocoder training device, the singing conversion device, the electronic device, and the storage medium used to execute the vocoder training method provided in this application. For the specific implementation process and technical effects, please refer to the relevant content of the vocoder training method and the singing conversion method mentioned above, which will not be repeated below.

[0102] Figure 8 This is a schematic diagram of the structure of a vocoder training device provided in an embodiment of this application, as shown below. Figure 8 As shown, the vocoder training device may include: The acquisition module 101 is used to acquire the standard vocal sample and the converted vocal sample corresponding to the sample song. The standard vocal sample and the converted vocal sample are the singing audio of the sample song with the same timbre. The converted vocal sample is the singing audio of the sample song with artifacts and high-pitched drift. The standard vocal sample is the standard singing audio of the sample song. Extraction module 102 is used to extract the pitch of the sample standard human voice to obtain the sample standard pitch features; The training module 103 is used to train the initial vocoder based on the sample standard human voice, the sample converted human voice, and the sample standard pitch features to obtain the vocoder.

[0103] Optionally, the acquisition module 101 is specifically used to separate the vocals and accompaniment in the sample song to obtain the sample separated vocals; to perform de-reverb and harmonization processing on the sample separated vocals to obtain the sample standard vocals; and to convert the sample standard vocals according to the timbre configuration of the sample standard vocals to obtain the sample converted vocals.

[0104] Optionally, the extraction module 102 is specifically used to extract sample pitch curves frame by frame from the sample standard human voice; perform formal transformation on the sample pitch curves to obtain sample pitch features; perform normalization processing on the sample pitch features to obtain normalized sample pitch features; and perform smoothing processing and note boundary alignment processing on the normalized sample pitch features to obtain the sample standard pitch features.

[0105] Optionally, the training module 103 is specifically used to use the fusion module in the initial vocoder to fuse the sample converted human voice and the sample standard pitch features to obtain sample fusion features; use the processing module in the initial vocoder to process the sample fusion features output by the fusion module to obtain sample predicted human voice; calculate multiple loss function values ​​based on the sample predicted human voice and the sample standard human voice; and update the model parameters of the initial vocoder based on the multiple loss function values ​​to obtain the vocoder.

[0106] Optionally, the training module 103 is specifically used to use the channel splicing module in the fusion module to splice the sample-converted human voice and the sample standard pitch features to obtain spliced ​​sample features; to perform an affine transformation on the spliced ​​sample features to obtain transformed sample features; and to use the gated residual module in the fusion module to fuse the spliced ​​sample features output by the channel splicing module and the sample-converted human voice to obtain the sample fusion features.

[0107] Optionally, the training module 103 is specifically used to calculate the multi-scale short-time Fourier transform reconstruction loss value, adversarial loss value, perceptual loss value, and pitch consistency loss value based on the human voice predicted from the sample and the standard human voice from the sample, respectively.

[0108] Figure 9 This is a schematic diagram of the structure of a singing voice conversion device provided in an embodiment of this application, as shown below. Figure 9 As shown, the singing voice conversion device includes: The separation module 201 is used to separate the vocals and accompaniment in the original song to obtain the accompaniment and the original vocal timbre. The conversion module 202 is used to convert the original timbre of the human voice to obtain the target timbre of the human voice. Processing module 203 is used to perform sound encoding processing on the target timbre human voice using a vocoder to obtain the target timbre standard human voice, wherein the vocoder is a vocoder obtained by using the vocoder training method described in any of the first aspects above. The generation module 204 is used to generate a target song based on the target timbre standard vocal and the accompaniment.

[0109] Optionally, the conversion module 202 is specifically used to perform de-reverb and de-chord processing on the original timbre vocal to obtain the processed original timbre vocal; and to perform timbre conversion on the processed original timbre vocal to obtain the target timbre vocal.

[0110] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0111] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0112] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 10 As shown, the electronic device includes: a processor 301 and a memory 302.

[0113] The memory 302 is used to store programs, and the processor 301 calls the programs stored in the memory 302 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described in detail here.

[0114] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0118] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of vocoder training, the method comprising: The method comprises: obtaining sample standard vocals and sample converted vocals corresponding to a sample song, wherein the sample standard vocals and the sample converted vocals are singing audio of the sample song with the same timbre, the sample converted vocals are singing audio of the sample song with artifacts and high pitch drift, and the sample standard vocals are standard singing audio of the sample song; extracting a pitch of the sample standard vocals to obtain sample standard pitch features; training an initial vocoder according to the sample standard vocals, the sample converted vocals and the sample standard pitch features to obtain a vocoder.

2. The method of claim 1, wherein, The obtaining of the sample standard vocals and the sample converted vocals corresponding to the sample song comprises: separating vocals and accompaniment in the sample song to obtain sample separated vocals; processing the sample separated vocals to obtain the sample standard vocals; converting the sample standard vocals according to the timbre configuration of the sample standard vocals to obtain the sample converted vocals.

3. The method of claim 1, wherein, The extracting of the pitch of the sample standard vocals to obtain the sample standard pitch features comprises: extracting a sample pitch curve from the sample standard vocals frame by frame; performing form conversion on the sample pitch curve to obtain sample pitch features; performing normalization processing on the sample pitch features to obtain normalized sample pitch features; performing smoothing processing and note boundary alignment processing on the normalized sample pitch features to obtain the sample standard pitch features.

4. The method of claim 1, wherein, The training of the initial vocoder according to the sample standard vocals, the sample converted vocals and the sample standard pitch features to obtain the vocoder comprises: using a fusion module in the initial vocoder to perform fusion processing on the sample converted vocals and the sample standard pitch features to obtain sample fusion features; using a processing module in the initial vocoder to process the sample fusion features output by the fusion module to obtain sample predicted vocals; calculating a plurality of loss function values according to the sample predicted vocals and the sample standard vocals; updating model parameters of the initial vocoder based on the plurality of loss function values to obtain the vocoder.

5. The method of claim 4, wherein, The using of the fusion module in the initial vocoder to perform fusion processing on the sample converted vocals and the sample standard pitch features to obtain the sample fusion features comprises: using a channel splicing module in the fusion module to splice the sample converted vocals and the sample standard pitch features to obtain spliced sample features; performing affine transformation on the spliced sample features to obtain transformed sample features; using a gated residual module in the fusion module to fuse the spliced sample features output by the channel splicing module and the sample converted vocals to obtain the sample fusion features.

6. The method of claim 4, wherein, The calculating of the plurality of loss function values according to the sample predicted vocals and the sample standard vocals comprises: According to the sample predicted human voice and the sample standard human voice, a multi-scale short-time Fourier transform reconstruction loss value, an adversarial loss value, a perceptual loss value and a pitch consistency loss value are respectively calculated.

7. A singing voice conversion method characterized by comprising: The method comprises: Separating the human voice and the accompaniment in the original song to obtain the accompaniment and the original timbre human voice; Performing timbre conversion on the original timbre human voice to obtain a target timbre human voice; Performing sound encoding processing on the target timbre human voice by using a vocoder to obtain a target timbre standard human voice, wherein the vocoder is a vocoder obtained by using the vocoder training method in any one of claims 1-6; Generating a target song according to the target timbre standard human voice and the accompaniment.

8. The method of claim 7, wherein, The method comprises: Performing de-reverberation and de-chorus processing on the original timbre human voice to obtain a processed original timbre human voice; Performing timbre conversion on the processed original timbre human voice to obtain the target timbre human voice.

9. An electronic device, comprising: Comprise: A memory and a processor, the memory stores a computer program executable by the processor, and the processor implements the method in any one of claims 1-8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the storage medium and is read and executed to implement the method in any one of claims 1-8.

Images