Zero-sample singing tone conversion method and system

Abstract

The invention relates to a zero-sample singing tone conversion method and system, and belongs to the technical field of digital audio processing and artificial intelligence, and the method specifically comprises the steps: extracting the content features, fundamental frequency features and tone embedding of a data set audio; a front-end encoder and a rear-end diffusion model are constructed successively and then preliminary training is carried out, a joint training strategy is adopted, end-to-end synchronous optimization is carried out by using data set audio data, and training of the front-end encoder and the rear-end diffusion model is completed; extracting content features and fundamental frequency features of a source audio and target timbre of a target audio, embedding and inputting the content features and the fundamental frequency features and the target timbre of the target audio into a trained front-end encoder for preliminary conversion, generating a preliminary conversion spectrum, then entering a rear-end diffusion model for double-path reverse iteration noise reduction processing, outputting spectrum components corresponding to two paths, adding elements one by one, and finally obtaining a final conversion spectrum; and the sound is converted into an audio waveform, thereby completing tone conversion processing of the singing sound. According to the method, the mixed training and the harmonic supervision are combined, so that the pitch accuracy and the tone fidelity of zero-sample singing tone conversion are improved.

Description

Technical Field

[0001] This invention belongs to the field of digital audio processing technology, specifically relating to a zero-sample singing timbre conversion method and system based on harmonic loss and fundamental frequency embedding. Background Technology

[0002] Speech timbre conversion is an artificial intelligence technology that aims to replace the timbre of a target speaker while preserving the content and prosodic features of the source language. Its core principle lies in deep learning. For example, patent document CN119673185A discloses a singing voice conversion model training method, a song timbre conversion method, and related products. The singing voice conversion model training method includes: extracting sample songs and matching vocal timbre description text; extracting timbre features of the sample songs using the audio encoding module in the singing voice conversion model, and extracting text features corresponding to the vocal timbre description text using the text encoding module in the singing voice conversion model; adjusting the model parameters of the singing voice conversion model based on the difference between the timbre features and the text features to obtain a trained singing voice conversion model; and using the trained singing voice conversion model to extract text features corresponding to the input timbre description text through the text encoding module, and obtaining the target timbre features based on the text features corresponding to the timbre description text, thus performing song timbre conversion. There is also a domestic patent document CN118782016A, which discloses a method and apparatus for fast conversion of singing voice timbre based on zero-shot sampling. It includes constructing a singing dataset containing dry human voice and lyrics text, constructing a singing voice feature decoupler containing a Hubert model and residual quantization codebook to extract the audio codebook index sequence of the dry human voice, and introducing a text encoder to extract the morpheme features and morpheme index sequence of the lyrics text. The singing voice feature decoupler is optimized by cross-prediction to improve the accuracy of speech content feature extraction. On this basis, pitch features and timbre features representing prosody are introduced. Adversarial training is used to enhance the quality of the synthesized human voice generated by the generator based on speech content features, pitch features, and timbre features, so as to realize the rapid conversion of the singing human voice timbre to the user's timbre. In the aforementioned patented technology, the training phase model relies on sample songs and precisely matched timbre description text, as well as fixed sample pronunciation information (pitch and phonemes). The conditions in the training and inference phases are inconsistent in all aspects, such as the fixed pronunciation conditions and the inconsistent pronunciation condition variables. This leads to insufficient generalization ability of the model and problems such as timbre distortion, pitch confusion, phoneme pronunciation distortion, and failure of material audio feature fusion during inference.

[0003] Therefore, the development of existing speech timbre conversion models faces two levels of technical challenges:

[0004] First, in zero-shot speech timbre conversion tasks, many models suffer from inconsistencies between training and inference conditions. During training, the model encounters timbre-matched data, while during zero-shot inference, it must handle timbre-mismatched conditions. This inconsistency limits the model's timbre conversion capabilities. Although existing research has attempted to address similar issues in other fields, the unique characteristics of singing signals in terms of pitch sensitivity and harmonic structure complexity prevent existing strategies from being directly applied to singing conversion tasks, often leading to severe sound quality degradation and pitch shift. Therefore, effectively addressing the inconsistency between training and inference conditions in the field of singing conversion remains a pressing technical challenge.

[0005] Secondly, new challenges arise when the model is applied to the specific domain of singing voice. Singing signals have extremely high requirements for the accuracy of the fundamental frequency and the integrity of the harmonic structure in the spectrum. Existing models designed for speech, due to a lack of targeted optimization, often produce singing voices with inaccurate pitch and degraded sound quality due to missing harmonics. Summary of the Invention

[0006] To simultaneously address the inconsistency between training and inference conditions in existing singing voice timbre conversion models, as well as the issues of inaccurate pitch and poor sound quality in singing voice timbre conversion tasks, this invention provides a zero-sample singing voice timbre conversion method, system, and computer equipment based on a hybrid training strategy and harmonic loss.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A zero-sample vocal timbre conversion method, the method comprising the following steps:

[0009] S1. Extract content features, fundamental frequency features, and timbre embeddings from the audio dataset;

[0010] S2. After constructing the front-end encoder and back-end diffusion model, initial training is performed. A joint training strategy is adopted, utilizing the content features, fundamental frequency features, and timbre embeddings of the dataset audio to synchronously optimize the initially trained front-end encoder and back-end diffusion model end-to-end, thereby improving their joint loss function L. total Minimize; repeat the reverse iteration until the joint loss function L is reached. total The training converges or reaches the preset number of training epochs, resulting in a well-trained front-end encoder and back-end diffusion model.

[0011] S3. Extract the content features and fundamental frequency features of the source audio and the target timbre of the target audio, embed them into the trained front-end encoder for preliminary conversion, and generate a preliminary conversion spectrum.

[0012] S4. Embed the preliminary conversion spectrum and target timbre from step S3 into the pre-trained back-end diffusion model for dual-path inverse iterative noise reduction processing, and output the spectral components corresponding to the two paths.

[0013] S5. Add the spectral components of S4 element by element and convert them into an audio waveform to complete the timbre conversion process of the singing voice.

[0014] Further specifying, the front-end encoder in step S2 is initially trained using the following method:

[0015] The content features, fundamental frequency features, and timbre embeddings of the dataset audio extracted in step S1 are used as initial inputs, and ordinary data batches matching the content features, fundamental frequency features, and timbre embeddings are loaded from them.

[0016] Convert the ordinary data batch into a mixed batch where the content features, fundamental frequency features, and timbre embeddings do not match;

[0017] Merge ordinary data batches and mixed batches to form a large batch set;

[0018] The front-end encoder generates a predicted Mel spectrum based on this large batch set, and obtains the total loss function of the front-end encoder by weighting the Mel reconstruction loss function and harmonic loss function, and then calculates the total loss of the front-end encoder.

[0019] Then, by adjusting the loss weight hyperparameters, the total loss of the front-end encoder is minimized using backpropagation, thus completing the initial training of the front-end encoder.

[0020] Further specifying, the preliminary training method for the front-end encoder in step S2 is as follows:

[0021] A data batch containing N audio samples is randomly selected from the audio dataset. For each audio sample in the data batch, the corresponding content features, fundamental frequency features, and timbre embeddings are extracted using the method described in step S1, thereby forming a set of N data samples with mutually matched features, denoted as the ordinary set B. normal ;

[0022] Define a random permutation function that operates on the index set {1, 2, ..., N}, and use the random permutation function to transform the ordinary set B. normal The data sample that is converted into a mixed batch containing N content features, fundamental frequency features, and mismatched timbre embeddings is called the mixture set B. mixed ;

[0023] The ordinary set B normal With the mixed set B mixed Perform a union operation to form the final large batch set B used for training. large :

[0024] B large =B normal ∪B mixed

[0025] The large batch set B large It contains 2N data samples;

[0026] Large batch set B large The input to the front-end encoder generates a predicted Mel spectrum for each sample;

[0027] Select the corresponding ordinary set B from the generated predicted Mel spectrum. normal The portion is compared with the target Mel spectrum, and the Mel reconstruction loss function and harmonic loss function are calculated respectively. The total front-end loss function of the front-end encoder is then constructed, and the total front-end loss L of the front-end encoder is calculated. fe ;

[0028] L fe =λ mel L mel +λ har L har

[0029] Among them, L fe This represents the total loss at the front end;

[0030] L mel Reconstruct the loss for Mel;

[0031] L har For harmonic loss;

[0032] λ mel and λ har These are the preset Mel reconstruction loss weight hyperparameters and harmonic loss weight hyperparameters, respectively.

[0033] Adjust the preset hyperparameters of Mel reconstruction loss and harmonic loss weights, and use the backpropagation algorithm to make the gradient of the total loss flow back to the front-end encoder, so that the total loss of the front-end encoder reaches the minimum, the iteration terminates, and the initial training of the front-end encoder is completed.

[0034] Further specifying, the initial training method for the backend diffusion model is as follows:

[0035] The preliminary source spectrum and preliminary filter spectrum of the front-end encoder output are randomly selected as complementary conditions and are input together with the timbre embedding into the dual-path score estimation network.

[0036] The diffusion process generates perturbation data and calculates the true scores, which drive the dual-path score estimation network to estimate the scores separately. The network parameters are then optimized collaboratively by minimizing the score matching loss, thus completing the initial training of the backend diffusion model.

[0037] Further specifying, the initial training method for the backend diffusion model is as follows:

[0038] Randomly select one set of complementary conditions from the two sets of outputs generated by the front-end encoder: if the conditions of the source fraction estimation network are selected from the preliminary source spectrum of the mixed set, then the conditions of the filter fraction estimation network are selected from the preliminary filter spectrum of the ordinary set, and vice versa, thus completing the construction of complementary settings.

[0039] A time point t is randomly selected from the continuous time interval [0, T], and a standard Gaussian noise z is also selected.

[0040] Let the target Mel spectrum M target Starting from the initial data x(0), the perturbation data x of the source path is generated using standard Gaussian noise z and the analytical solution of the forward SDE. src (t) and the perturbation data x of the filter path flt (t);

[0041] Based on the perturbation data x from the source path src (t) and the perturbation data x of the filter path flt Given x(t) and the initial data x(0), calculate the true fractional function at that moment. and

[0042]

[0043] Where σ t It is a time-dependent noise scale;

[0044] Source Score Estimation Network Dec src and filter fraction estimation network Dec flt Receive disturbance data x from the source path respectively src (t) and the perturbation data x of the filter path flt After embedding the t (t), time t, and timbre with the dataset audio, the estimated values ​​of the true scores at that moment are estimated according to the complementary settings.

[0045] Through backpropagation, the score matching loss L is used. diff The resulting gradient iteratively updates the weights of the source score estimation network and the filter score estimation network until the score matching loss L... diff Minimum, followed by initial training of the downstream diffusion model.

[0046] Further specifying, step S3 specifically involves: extracting the source content features C of the source audio. src Source fundamental frequency characteristics P srcAnd the target audio's target timbre embedding spk tgt The input is fed into the trained front-end encoder Enc. FE In this process, the source encoder and filter encoder in the front-end encoder are used for parallel path processing. The source encoder receives the source fundamental frequency feature P. src embed the target tone into spk tgt Output the preliminary source spectrum after initial conversion. Filter encoder receives C src embed the target tone into spk tgt Output the preliminary filter spectrum after initial conversion.

[0047] Further specifying, step S4 specifically includes:

[0048] Dec utilizes two parallel source score estimation networks in a back-end diffusion model src and filter fraction estimation network Dec flt High-quality spectral components are generated from random noise through an iterative reverse sampling process.

[0049] The specific method for generating high-quality spectral components from random noise through an iterative inverse sampling process is as follows:

[0050] Based on the selected sampler algorithm, sample from the set number of inference steps N. infer Initially, the inverse sampling process is executed iteratively until time step 0. In each sampling step, the sampler calls the source score estimation network Dec. src It provides the noisy input of the current path, the current sampling step number, and the preliminary source spectrum as conditions. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step; simultaneously, the sampler calls the filter fractional estimation network Dec. flt It provides the noisy input of the current path, the current sampling step number, and the preliminary filter spectrum as conditions. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step; after inference sampling steps N infer After step-by-step iterative sampling, the final source components of the two path outputs are obtained respectively. and final filter components

[0051] Further specifying, S5 specifically refers to:

[0052] Dec src and filter fraction estimation network Dec flt The final source component of the output and final filter components Element-wise addition is performed to obtain a high-fidelity Mel spectrum, which is then input into a vocoder to convert it into a playable audio waveform, thus completing the timbre conversion process of the singing voice.

[0053] Further specifying, the specific method for extracting the content features, fundamental frequency features, and timbre embedding of the dataset audio in step S1 is as follows:

[0054] S11. Load the audio dataset D and extract audio sample A from the audio dataset D. raw ;

[0055] S12, For audio sample A raw Data augmentation is performed by applying random transformations of pitch and formants, and content features C are extracted from the augmented audio using a content encoder;

[0056] S13. Using a pitch extraction algorithm to extract from audio sample A raw The fundamental frequency curves of each frame are obtained, and the fundamental frequency value f0(t) of each frame is converted into a more uniform scale in perception to obtain the fundamental frequency feature P.

[0057] S14. Using the audio preprocessing module to process audio sample A raw Extract the target Mel spectrum M target ;

[0058] S15, Using a timbre encoder to extract audio sample A raw Extract the timbre features used to represent the speaker's identity and embed them as timbre embeddings (spk).

[0059] The present invention also provides a zero-sample singing timbre conversion system, comprising:

[0060] The feature extraction module is used to extract content features, fundamental frequency features, and timbre embeddings from the audio data set.

[0061] Model training module: Used for initial training after building the front-end encoder and back-end diffusion model sequentially. It employs a joint training strategy, utilizing the content features, fundamental frequency features, and timbre embeddings of the dataset's audio to perform end-to-end synchronous optimization of the initially trained front-end encoder and back-end diffusion model, improving their joint loss function L. total Minimize; repeat the reverse iteration until the joint loss function L is reached. total The training converges or reaches the preset number of training epochs, resulting in a well-trained front-end encoder and back-end diffusion model.

[0062] The front-end encoder processing module is used to embed the extracted content features, fundamental frequency features, and target timbre of the target audio into the trained front-end encoder for preliminary conversion, generating a preliminary conversion spectrum.

[0063] The back-end diffusion model processing module is used to embed the initial converted spectrum and the target timbre into the input trained back-end diffusion model for dual-path inverse iterative noise reduction processing, and output the spectral components corresponding to the two paths.

[0064] The synthesis and conversion module is used to add the spectral components output by the back-end diffusion model processing module element by element and convert them into audio waveforms to complete the timbre conversion processing of the singing voice.

[0065] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0066] The singing timbre conversion method provided by this invention, based on an architecture including a front-end encoder and a back-end diffusion model, employs a hybrid training strategy to address the inconsistency between training and inference conditions. This strategy introduces both timbre-matched and timbre-mismatched training samples during training, allowing the model to pre-learn to handle inconsistent conditions and achieve zero-sample conversion capability. Furthermore, to address the pitch and timbre issues in singing timbre conversion, random transformations of pitch and formants are applied to the original audio as data augmentation before content feature extraction, helping the model initially learn content representations independent of timbre and pitch. In the model's input, fundamental frequency features are transformed and used as independent conditional inputs to enhance the model's precise control over the singing melody. During the training of the front-end encoder, a harmonic loss function is added to its loss function. This loss function, by supervising the harmonic positions in the singing spectrum, improves the model's ability to reconstruct harmonic structures, thereby improving the timbre quality of the generated singing. Attached Figure Description

[0067] Figure 1 Flowchart of the zero-sample vocal timbre conversion method.

[0068] Figure 2 A schematic diagram illustrating the construction of training data for the front-end encoder.

[0069] Figure 3 This is a schematic diagram of the spectral energy distribution of a single frame.

[0070] Figure 4 A schematic diagram of the vocal timbre conversion system for singing. Detailed Implementation

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0073] The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0074] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0075] The following is a definition of the terminology used in this application:

[0076] Front-end Encoder: In this invention, it refers to a network module that receives content features, fundamental frequency features, and timbre embeddings as input and outputs one or more preliminary conversion spectra (e.g., preliminary source spectra and preliminary filter spectra).

[0077] Back-end Diffusion Model: In this invention, it refers to a generation model that receives the initial conversion spectrum generated by the front-end encoder as a condition and generates the final high-fidelity Mel spectrum from an initial noise through an iterative denoising process.

[0078] Timbre Embedding: In this invention, it refers to a fixed-dimensional vector representing the timbre features of an audio dataset or target audio, extracted from the Mel spectrum of the audio dataset or target audio by a timbre encoder.

[0079] Mixed Training Strategy: In this invention, it specifically refers to a strategy for training a backend diffusion model. This strategy, at each step of the training process, introduces both conditional samples that match the target timbre and conditional samples that do not match the target timbre, in order to jointly optimize the model parameters.

[0080] Harmonic Loss: In this invention, it specifically refers to a loss function. It is calculated as follows: First, based on the fundamental frequency characteristics of the input, the positions of multiple frequency bands corresponding to the fundamental frequency and its harmonics are determined in the target Mel spectrum; then, only the differences between the predicted Mel spectrum generated by the front-end encoder during the training phase and the target Mel spectrum at these determined frequency band positions are calculated.

[0081] Patent full text variable definition table:

[0082] 1. Data Processing and Feature Extraction (S1)

[0083]

[0084]

[0085] 2. Front-end encoder training (S201)

[0086]

[0087]

[0088] 3. Backend diffusion model training (S202)

[0089]

[0090]

[0091] 4. Joint Training (S203)

[0092]

[0093] 5. Transition Phase (S3, S4, S5)

[0094]

[0095] The following provides an exemplary application environment for this application. Figure 1 This is a flowchart of a zero-sample singing timbre conversion method according to an embodiment of the present invention. Figure 2 A schematic diagram illustrating the construction of training data for the front-end encoder. Figure 3 This is a schematic diagram of the spectral energy distribution of a single frame. The zero-sample vocal timbre conversion method in this embodiment can be implemented by the following steps:

[0096] S1. Extract content features, fundamental frequency features, and timbre embeddings from the audio dataset;

[0097] The specific extraction operation method is as follows:

[0098] S11. Load the audio dataset D and extract audio sample A from the audio dataset D. ras ;

[0099] S12, For audio sample A raw Data augmentation is performed by applying random transformations of pitch and formants, and content features C are extracted from the augmented audio using a content encoder;

[0100] S13. Using a pitch extraction algorithm to extract from audio sample A ras The fundamental frequency curves of each frame are obtained, and the fundamental frequency value f0(t) of each frame is converted into a more uniform scale in perception to obtain the fundamental frequency feature P.

[0101] S14. Using the audio preprocessing module from A ras Extract the target Mel spectrum M target ;

[0102] S15, Using a timbre encoder to extract audio sample A raw Extract the timbre features used to represent the speaker's identity and embed them as timbre embeddings (spk).

[0103] Specifically:

[0104] Content Feature C: First, analyze the original audio sample A. raw Random transformations of pitch and formants are applied; for example, in one embodiment, the pitch may be randomly shifted within a range of ±3 semitones, and the formants may be randomly shifted within a range of ±20% of the frequency range, as data augmentation. The augmented audio is then input into a pre-trained content encoder (such as Wav2Vec) to obtain content features C.

[0105] Fundamental frequency feature P: First, a pitch extraction algorithm (such as YAAPT) is used to extract the fundamental frequency feature P from the original audio sample A. raw The fundamental frequency (F0) curves are obtained frame by frame, with the numerical unit of the curve being Hertz (Hz). Considering that human hearing perceives pitch not on a linear scale but rather on a near-logarithmic scale, this invention performs a perceptual scale transformation on the extracted original fundamental frequency curves to enable the model to learn and control melody more effectively. Specifically, the fundamental frequency value f0(t) (in Hz) of each frame is transformed into a more perceptually uniform scale through a nonlinear mapping function. For example, it can be transformed into a Mel scale or a purely logarithmic scale.

[0106] In a specific implementation of the present invention, the transformation includes the following steps: first, applying the Mel scale transformation formula to the non-zero fundamental frequency value f0(t) in each time frame to obtain its corresponding value m on the Mel scale. f0 (t), and then the continuous Mel-scale fundamental frequency sequence m f0 (t) is quantized to obtain a set of discrete integer indices. In one embodiment, the frequency range of the Mel scale can be quantized into 192 discrete levels, but this is not the only limitation; the number of quantization levels can also be selected according to the required pitch resolution, for example, in the range of 128 to 1024. This final set of discrete fundamental frequency indexes serves as the fundamental frequency feature P provided to downstream modules of the model (such as the source encoder). Through this transformation, physically proportional pitches (such as octaves) will be approximately represented as arithmetic progressions in the feature space, which significantly reduces the difficulty for the neural network to learn melody patterns.

[0107] Target Mel spectrum M target Using an audio preprocessing module to process raw audio sample A raw Extraction. The conversion process includes: first, framing and windowing the original audio sample signal; then, performing a Fast Fourier Transform (FFT) on each frame to obtain its power spectrum; subsequently, filtering the power spectrum through a Mel filter bank; and finally, performing a logarithmic operation on the output energy of each Mel filter to obtain the target Mel spectrum M. target .

[0108] Phonogram embedding (SPK): This involves extracting timbre features from audio using a timbre encoder (such as a model based on context-aware masking and aggregation networks like CAM++) to represent the speaker's identity. This can also be referred to as speaker embedding below. This model can efficiently extract deep features from raw audio samples that stably represent the speaker's identity, serving as the timbre embedding (SPK).

[0109] S2. After constructing the front-end encoder and back-end diffusion model, initial training is performed. A joint training strategy is adopted, utilizing the content features, fundamental frequency features, and timbre embeddings of the dataset audio to synchronously optimize the initially trained front-end encoder and back-end diffusion model end-to-end, thereby improving their joint loss function L. total Minimize; repeat the reverse iteration until the joint loss function L is reached. tota The training converges or reaches the preset number of training epochs, resulting in a well-trained front-end encoder and back-end diffusion model.

[0110] Specifically:

[0111] S201: Initial Training of the Front-End Encoder

[0112] The front-end encoder Enc FE A dual-path parallel architecture is adopted, consisting of a source encoder and a filter encoder. The training steps of the front-end encoder include three stages: construction of training data, forward propagation of the model, and parameter optimization based on the loss function.

[0113] The content features, fundamental frequency features, and timbre embeddings of the dataset audio extracted in step S1 are used as initial inputs, and ordinary data batches matching the content features, fundamental frequency features, and timbre embeddings are loaded from them.

[0114] Convert the ordinary data batch into a mixed batch where the content features, fundamental frequency features, and timbre embeddings do not match;

[0115] Merge ordinary data batches and mixed batches to form a large batch set;

[0116] Figure 2 This is a schematic diagram of training data construction. The process of constructing training batches is as follows:

[0117] Randomly select N original audio samples; for each audio sample, extract its corresponding content features, fundamental frequency features, and timbre embedding using the method in step S1, and combine these three features from the same audio sample into a data sample, thereby constructing a general set B containing N mutually matching data samples. normal

[0118] The ordinary set is expressed as follows:

[0119]

[0120] Here, i is the sample index, and C i ,P i , and spk i These represent the content features, fundamental frequency features, and timbre embedding of the i-th sample, respectively.

[0121] Next, a random permutation function π is defined to operate on the index set {1,2,...,N}. Using this random permutation function, the ordinary set is transformed into a mixed batch containing N data samples with mismatched content features, fundamental frequency features, and timbre embeddings, denoted as the mixed set B. mixed Its form is:

[0122]

[0123] In this hybrid set, the content features and fundamental frequency features of each sample remain consistent with its original index i, but its timbre embedding comes from the sample corresponding to the index π(i) mapped by the permutation function π. Finally, the ordinary batch set and the hybrid batch set are joined to form a final large batch set B used for training. large :

[0124] B large =B normal ∪B mixed

[0125] The final large batch set B large It contains 2N data samples.

[0126] The front-end encoder generates a predicted Mel spectrum based on the aforementioned large batch set, and obtains the total front-end loss function of the front-end encoder by weighting the Mel reconstruction loss function and harmonic loss function, thereby calculating the total front-end loss of the front-end encoder; specifically:

[0127] This large batch of data B large Input to the front-end encoder. This process generates a corresponding output spectrum for each sample in a large batch. Specifically:

[0128] For subset B of the normal batch normal Each sample (C) i P i spk i The front-end encoder generates a set of corresponding matching spectra, at which point the source encoder receives P. i and spk i Output source content spectrum (normal) The filter encoder receives C i and spk i Output filter content spectrum (normal)

[0129] For the subset B of the mixed batch mixed Each sample (C) i ,P i spk π(i) The front-end encoder generates a set of corresponding mismatched spectra, at which point the source encoder receives P. i and spk π(i) Output source content spectrum (mixed) Filter encoder receives C i and spk π(i) Output filter content spectrum (mixed) Therefore, a complete forward propagation will produce two sets of source content spectra and two sets of filter content spectra, corresponding to the inputs of the normal batch and the mixed batch, respectively.

[0130] The total front-end loss function L of the front-end encoder fe Reconstruction loss L by Mel mel Harmonic loss L har The loss is calculated using a weighted summation. This loss only applies to subset B of the ordinary batch within the larger batch. normal The calculation is performed on that portion of the data. Its definition is as follows:

[0131] L fe =λ mel L mel +λ har L har

[0132] Where, λ mel and λ har This is a preset loss weight hyperparameter. In one embodiment of the present invention, the weight hyperparameter λ... mel and λ har These values ​​are set to constant values ​​throughout the training process. These values ​​can be determined through experimental tuning to optimize the model's performance on a specific task or dataset. In a specific implementation of this invention, the weights λ... mel The value of λ is set to 1.0, and the weight λ har The value is set to 0.5. This configuration ensures that the generated spectrum and the target spectrum maintain overall structural consistency (by L). mel While constraining (the harmonic structure that determines pitch perception, composed of L...), har Additional supervision is imposed by the constraints. Those skilled in the art will understand that the specific values ​​of the weight hyperparameters are not the only limitation of this invention.

[0133] By adjusting the loss weight hyperparameters, the total loss of the front-end encoder is minimized using backpropagation. Specifically, without departing from the core idea of ​​this invention, the loss of λ can be adjusted according to different model architectures, dataset characteristics, or optimization objectives. mel and λ har Adjust the value to achieve the best results.

[0134] The Mel reconstruction loss (L) mel ) used to calculate the predicted Mel spectrum M generated by the front-end encoder gen (by B) normal generated and (sum of) and target Mel spectrum M target The mean absolute error between them. Its calculation formula is:

[0135]

[0136] Where n and t are the frequency and time dimensions of the Mel spectrum, respectively, n∈[1, N], t∈[1, T]. The harmonic loss L har Used to calculate M gen With M target The calculation principle for the average absolute error at the harmonic position determined by the fundamental frequency (F0) can be referred to [reference needed]. Figure 3 To understand this, the calculation method includes the following steps: First, determine the harmonic frequency index. The harmonic frequency index refers to the index based on the fundamental frequency profile F0(t) corresponding to the target audio frequency and a set of harmonic integers k = 1, 2, ..., N. har A series of physical harmonic frequencies f were calculated. k (t) = k·F0(t). Where, N har The highest harmonic order N is considered. In a specific implementation of the present invention, the highest harmonic order N is... har The value is 15. Those skilled in the art will understand that this value was chosen to cover the major harmonic components that play a decisive role in timbre perception in most singing styles; the harmonic richness of a voice is closely related to the singer's vocal technique and singing style. N har Setting N to 15 effectively models the rich harmonic structure produced by professionally trained singers, thus ensuring the fullness and realism of the generated vocal timbre. However, this invention is not limited to this specific value; N... har Alternatively, other values ​​can be selected within a reasonable range, depending on the specific application scenario or singing style, such as any integer between 5 and 30.

[0137] Subsequently, the physical harmonic frequency f is mapped to an index using a frequency-to-index mapping function "Mel"(·). k (t) is converted to a discrete index m on the Mel spectrum. k (t)=Mel(f k (t)). The mapping function "Mel"(·) converts a continuous physical frequency value in Hertz into an integer index of its corresponding frequency bin in a discrete Mel spectrogram. This mapping process is based on the nonlinear perception of frequency by the human ear, i.e., the Mel scale. In the Mel scale, equidistant pitch differences perceived by the human ear correspond to unequal physical frequency differences, specifically manifested as sensitivity in the low-frequency region and insensitivity in the high-frequency region.

[0138] Specifically, the implementation of the mapping function "Mel"(f) involves the following two steps:

[0139] Step 1: Convert the physical frequency f (in Hz) to the mezzanine frequency m (in Mel). This conversion can be achieved using formulas well-known in the art, for example:

[0140]

[0141] Step 2: Linearly map the Mel frequency m to the discrete frequency index of the Mel spectrogram. Since the frequency axis of the Mel spectrogram is uniformly divided on the Mel scale, a simple scaling conversion can be used to determine which frequency range the Mel frequency m falls into, thus obtaining its corresponding integer index. Through this mapping function, the present invention can accurately locate the harmonic frequencies of the physical world to specific positions in the Mel spectrum representation that better conforms to human auditory perception, thereby achieving effective supervision of the harmonic structure. Those skilled in the art will understand that the perceptual scale transformation of the fundamental frequency feature in step S1 can also be implemented using the mapping function Mel(·) here to ensure the consistency of the entire model on the frequency perceptual scale.

[0142] The harmonic loss L har The calculation formula is:

[0143]

[0144] Wherein, S(t, m) k (t) represents the spectrum at time t, harmonic index m k The energy value at point (t), P valid It represents the set of all valid time frames and harmonic pairs. Figure 4 The calculation principle of the aforementioned harmonic loss is intuitively illustrated using a schematic diagram of the energy distribution of a single frame spectrum. In this diagram, the horizontal axis represents the frequency range of the spectrum, and the vertical axis represents the energy. Two different fill patterns of bars schematically represent the target Mel spectrum (M). target )) and predicted Mel spectrum (M gen The diagram illustrates the positions of each harmonic, determined by the fundamental frequency F0 and its harmonics. The core mechanism of this loss function—calculating the energy difference only at these specific harmonic positions—is clearly demonstrated by the bidirectional arrows and formula annotations in the diagram.

[0145] S202: Preliminary Training of the Back-End Diffusion Model

[0146] The backend diffusion model comprises two parallel score estimation networks: the source score estimation network Dec src and filter fraction estimation network Dec flt Its training process is based on the principle of score matching, which enables the model to master the ability to generate and transform data by learning the log probability density gradient (i.e., the score) of the data distribution during continuous-time perturbation. This process specifically includes the following steps:

[0147] The preliminary source spectrum and preliminary filter spectrum of the front-end encoder output are randomly selected as complementary conditions and are input together with the timbre embedding into the dual-path score estimation network.

[0148] The diffusion process generates perturbation data and calculates the true scores, which drive the dual-path score estimation network to estimate the scores separately. The network parameters are then optimized collaboratively by minimizing the score matching loss, thus completing the initial training of the backend diffusion model.

[0149] Specifically:

[0150] First, construct complementary hybrid training conditions:

[0151] From the two sets of outputs generated by the front-end encoder (normal set output and hybrid set output), a set of complementary conditions is randomly selected: if the conditions of the source fraction estimation network are selected from the preliminary source spectrum of the hybrid set, then the conditions of the filter fraction estimation network are selected from the preliminary filter spectrum of the normal set; otherwise, if the conditions of the source fraction estimation network are selected from the preliminary source spectrum of the normal set, then the conditions of the filter fraction estimation network are selected from the preliminary filter spectrum of the hybrid set, thus completing the construction of complementary settings.

[0152] In each training step, from the two sets of outputs generated by the front-end encoder S102, a set of "complementary" conditions is randomly selected for the two score networks. If the source score estimation network is Dec... src Preliminary source spectrum selected (Timbre mismatch) In this case, a filter fraction estimation network (Dec) must be used. flt Select the initial filter spectrum (Timbre matching), and vice versa. We use this step for the source score estimation network Dec src and filter fraction estimation network Dec flt The selected conditions are denoted as source path conditions. and filter path conditions This complementary setup ensures that the model can learn both high-fidelity reconstruction and zero-shot transformation simultaneously in each update.

[0153] Next, training data at different noise levels is needed for the dual-path score estimation network. This process is accomplished through a controlled random perturbation mechanism. In this invention, this perturbation mechanism is mathematically defined by a forward stochastic differential equation (SDE), which precisely describes the process of clear data continuously transforming into pure noise. To generate a training instance, this process specifically includes the following steps:

[0154] Step 1: Randomly sample a time point t from the continuous time interval [0, T] and sample a standard Gaussian noise z. In a typical variance-preserving (VP) SDE setting, T is usually set to 1.0.

[0155] Step 2: Let the target Mel spectrum M target Let x(0) be the initial data. Starting from the initial data x(0), two sets of perturbation data are generated using standard Gaussian noise z and the analytical solution of the forward SDE, namely the perturbation data x of the source path. src (t) and the perturbation data x of the filter path flt (t).

[0156] Specifically, the perturbation data of the source path is denoted as x. src (t), which consists of initial data x(0) and source path conditions. And standard Gaussian noise z-weighted; the perturbation data of the filter path is denoted as x. flt (t), which consists of initial data x(0) and filter path conditions. And it is composed of standard Gaussian noise z-weighted. The weights of these components are all functions of a preset time t. Taking the source path as an example, the weights are set such that when time t approaches 0, the perturbation data x of the source path... src The composition of (t) is almost entirely composed of the target Mel spectrum M target Dominant; and as time t increases and approaches its maximum value T, the target Mel spectrum M target The weight gradually decreases, source path condition The weights of the filter path and the Gaussian noise z increase simultaneously. ftr (t) also changes with t.

[0157] Step 3: Based on the perturbation data x of the source path sampled at time t src (t) and the perturbation data x of the filter path flt Given x(t) and the initial data x(0), the true fractional function at that moment can be calculated, denoted as: and

[0158] Mathematically, this fractional function represents the logarithmic probability density gradient of the data distribution. Intuitively, it guides the denoising process by indicating a direction that moves noisy data towards more likely (i.e., clearer) data points. It's worth noting that in many implementations, such as when using variance-preserving (VP) SDE, this true fractional function has a simple proportional relationship with the added random noise:

[0159]

[0160] Where σ tIt is a noise scale that is time-dependent. This characteristic greatly simplifies the training process because it transforms the problem of calculating a complex log probability density gradient into a simple algebraic operation that only requires knowledge of the added noise z and time t.

[0161] Source Score Estimation Network Dec src and filter fraction estimation network Dec flt The task is to accurately estimate the true score at a given moment, given the perturbed data and corresponding conditions. This is the source score estimation network Dec. src Received data x after disturbance src (t), time t, and timbre embeddings with the dataset audio, source score estimation network Dec src Output an estimate of the true score. Filter score estimation network Dec flt Perform a similar operation to receive the perturbation data x from the filter path. ftr The algorithm takes the timbre embeddings of the dataset audio (t), time t, and the timbre of the data, and outputs an estimate of the true score.

[0162] To optimize the two score estimation networks mentioned above, it is necessary to introduce a calculation of the score matching loss L. diff The loss function aims to minimize the expected gap between the model-estimated score and the true score, and is in the form of a weighted mean squared error:

[0163]

[0164] in, Representing the expectation of time, initial data, and perturbation data, λ(t) is a time-dependent positive weighting function used to balance the training contributions under different noise levels. This invention covers different configurations of this weighting function λ(t).

[0165] In one implementation, λ(t) can be set to a constant value, for example, 1. This configuration treats the accuracy of the fractional estimation as equally important in all diffusion stages, constituting a simple and effective benchmark implementation. To improve the model's performance in the region with extremely low signal-to-noise ratios (corresponding to the initial stage of the inverse denoising process), λ(t) can be specifically set as a function related to the noise schedule in the diffusion process. The noise schedule defines the variance of the noise component added to the signal at any time t. In a specific implementation of the invention, if the diffusion process is defined by variance-preserving (VP) SDE, its noise schedule function can be derived from the signal scale α. t and noise scale σ tIn this case, a particularly effective weighting function λ(t) is set as the reciprocal of the signal-to-noise ratio function, i.e. This weighting scheme forces the model to invest more learning resources in accurately estimating scores when the signal is almost completely submerged in noise, which is crucial for the eventual ability to generate high-fidelity outputs from pure random noise.

[0166] The loss function L mentioned above diff The ultimate goal is to enable the two score estimation networks to achieve high-fidelity reconstruction and zero-sample timbre conversion. This is under the condition of the source path. When the data comes from a normal batch (timbre matching), the source score estimation network learns an accurate estimate of the data distribution scores under timbre matching conditions, corresponding to high-fidelity reconstruction capability. When the filter path conditions... When the timbre comes from a mixed batch (timbre mismatch), the filter score estimation network must rely on the target timbre embedding spk to estimate the correct score, thereby learning the ability to perform zero-sample timbre conversion.

[0167] Through backpropagation, the score matching loss L is used. diff The resulting gradient iteratively updates the weights of the source score estimation network and the filter score estimation network until the score matching loss L... diff Minimum, followed by initial training of the downstream diffusion model.

[0168] S203: Joint Optimization

[0169] A joint training strategy is adopted to perform end-to-end synchronous optimization on the initially trained front-end encoder and back-end diffusion model, so that their joint loss function L total Minimize; repeat the reverse iteration until the joint loss function L is reached. total The training converges or reaches the preset number of training epochs, resulting in a trained front-end encoder and a back-end diffusion model.

[0170] Specifically:

[0171] This invention employs a joint training strategy to perform end-to-end synchronous optimization of the front-end encoder and the back-end diffusion model. This step aims to ensure that all components of the model work collaboratively to achieve the final optimization goal. The specific details are as follows:

[0172] The total loss function defines the objective of model optimization, and is described as follows: It calculates the loss function L of the front-end encoder. fe (Defined in S102) and the loss function L of the back-end diffusion model diff(Defined in S103) Perform weighted summation to form the total loss function L of the entire model. total Its mathematical expression is:

[0173] L total =L fe +α·L diff

[0174] Among them, L fe =λ mel L mel +λ har L har Weighting coefficients α and λ mel and λ har These are preset hyperparameters used to balance the contributions of different loss terms to model parameter updates. In the specific implementation of this invention, the weight α is set to 1.0, and λ... mel Set to 1.0, λ har The weight α is set to 0.5 to ensure a good balance between learning the basic spectral structure, harmonic details, and diffusion generation capabilities. To balance the reconstruction task of the front-end encoder with the generation task of the back-end diffusion model, the weight α is typically set in the range of 0.5 to 2.0.

[0175] After defining the loss function, gradient calculation and backpropagation are also required, as described below: In each training iteration step, the total loss function L is calculated. total The gradient is calculated relative to all trainable parameters of the model (including the parameters of the front-end encoder and the back-end diffusion model). The calculated gradient is then propagated to each layer of the model via backpropagation.

[0176] S204: Reverse Iteration

[0177] By repeatedly executing the training loops described in S201 to S203, until the total loss function L is reached... total The model converges or reaches the preset number of training cycles, ultimately resulting in a fully optimized model capable of performing high-quality zero-shot vocal timbre conversion.

[0178] This invention uses a gradient-based optimizer to update all trainable parameters of the model in order to minimize the total loss function L. total The AdamW optimizer can be used. AdamW is a variant of the Adam optimizer that improves the handling of weight decay, effectively preventing model overfitting and exhibiting better convergence and generalization capabilities in many deep learning tasks.

[0179] S3: Extract the content features and fundamental frequency features of the source audio and the target timbre of the target audio, embed them into the trained front-end encoder for preliminary conversion, and generate a preliminary conversion spectrum;

[0180] The purpose of this step is to perform preliminary conversion using the front-end encoder trained by S2, that is, to process the source audio and target audio according to the extraction and processing method of S1, and extract the source content features C of the obtained source audio. src Source fundamental frequency characteristics P src And the target audio's target timbre embedding spk tgt The extracted source audio content features C src Source fundamental frequency characteristics P sr

[0181] and the target audio's timbre embedding spk tgt Input to the trained front-end encoder Enc FE In this process, two parallel paths of the front-end encoder are used for processing, and the source encoder receives the source fundamental frequency feature P. src embed the target tone into spk tgt Output the preliminary source spectrum after initial conversion. Filter encoder received source content characteristics C src embed the target tone into spk tgt Output the preliminary filter spectrum after initial conversion.

[0182] S4. Embed the initial conversion spectrum of S3 and the target timbre into the input trained back-end diffusion model for dual-path inverse iterative noise reduction processing, and output the spectral components corresponding to the two paths.

[0183] Specifically:

[0184] Dec utilizes two parallel source score estimation networks in a back-end diffusion model src and filter fraction estimation network Dec flt High-quality spectral components are generated from random noise through an iterative reverse sampling process.

[0185] The iterative reverse sampling process specifically involves sampling from a predetermined total number of inference steps N according to the selected sampler algorithm. infer Initially, the inverse sampling process is executed iteratively until time step 0. In each sampling step, the sampler calls the source score estimation network Dec. src It provides the noisy input of the current path, the current sampling step number, and the source path conditions. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step; simultaneously, the sampler calls the filter fractional estimation network Dec.flt It provides the noisy input of the current path, the current sampling step, and the filter path conditions. embed the target tone into spk tgt To calculate the spectral estimate for the next sampling step; after N infer After step-by-step iterative sampling, two path output spectral components are obtained, which are the final source components. and final filter components

[0186] To elaborate further, the iterative backsampling process described above consists of the following steps:

[0187] The first step is to select an efficient inverse sampling algorithm, such as a solver based on stochastic differential equations (SDEs) or a solver based on probabilistic flow ordinary differential equations (PF-ODEs). Simultaneously, a total sampling number N for this inference is set. infer It should be noted that the number of inference sampling steps N here... infer Unlike the continuous-time training process in Example 1, this is the number of discretization steps used to numerically solve the aforementioned continuous-time inverse SDE or ODE.

[0188] N infer is a user-configurable hyperparameter used to balance generation quality and inference speed. For higher N... infer (For example, 100-200 steps) will result in finer sampling, usually producing higher sound quality, but the computational cost will also increase accordingly, leading to slower inference speed. Lower N infer (For example, 10-50 steps) can significantly speed up reasoning, but may compromise on sound quality details.

[0189] The improvement in sound quality does not come with N infer The improvement in sound quality increases indefinitely, but after a certain number of steps, it tends to saturate, reaching a performance ceiling. Therefore, users can choose a suitable N based on their specific application scenario. infer value.

[0190] The second step involves generating two independent standard Gaussian noise maps with dimensions matching the target spectrum, which will be used as the initial noise for the source path. and initial noise of the filter path

[0191] The third step is to use the sampler algorithm selected in the first step, starting from the set total number of steps N. infer Initially, the inverse sampling process is performed iteratively until time step 0. In each sampling step, for the source score estimation network (Dec... srcFor example, the sampler calls the source score estimation network Dec. src It provides the noisy input of the current path, the current sampling step number, and the preliminary source spectrum. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step. For the filter fractional estimation network (Dec... flt For example, the sampler calls the filter score estimation network Dec flt It provides the noisy input of the current path, the current sampling step number, and the preliminary filter spectrum. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step.

[0192] The final step, after N infer After iterative sampling, the clean final source components of the two paths are obtained respectively. and final filter components

[0193] S5: The spectral components of S4 are the final source components. and final filter components Element-wise addition is performed and converted into an audio waveform to complete the timbre conversion of the singing voice. This involves the source score estimation network Dec. src and filter fraction estimation network Dec flt The final source component of the output and final filter components Element-wise addition is performed to obtain a high-fidelity Mel spectrum, which is then input into a vocoder to convert it into a playable audio waveform, thus completing the timbre conversion process of the singing voice.

[0194] Specifically:

[0195] Dec, the two source score estimation networks mentioned above src and filter fraction estimation network Dec flt The final source component of the final output and final filter components Element-wise addition is performed to obtain the final high-fidelity Mel spectrum with the target timbre. This high-fidelity Mel spectrum is then input into a pre-trained vocoder to convert it into the final playable audio waveform.

[0196] Furthermore, the vocoder is a non-essential component for implementing this invention, and its specific implementation should not limit the scope of protection of this invention. For example, a vocoder based on a generative adversarial network (GAN), such as HiFiGAN or BigVGAN, can be used; other types of vocoders can also be used. Any technique capable of converting a Mel spectrum into an audio waveform falls within the scope of this invention.

[0197] Specifically, the front-end encoder of the present invention adopts a dual-path parallel structure, including a source encoder and a filter encoder. In one specific embodiment, the source encoder and the filter encoder can be implemented by a deep neural network (e.g., WaveNet network) containing multiple layers of dilated convolutions.

[0198] This invention provides a zero-sample vocal timbre conversion system, see [link to related document]. Figure 4 ,include:

[0199] The feature extraction module is used to extract content features, fundamental frequency features, and timbre embeddings from the audio data set.

[0200] Model training module: Used for initial training after building the front-end encoder and back-end diffusion model sequentially. It employs a joint training strategy, utilizing the content features, fundamental frequency features, and timbre embeddings of the dataset's audio to perform end-to-end synchronous optimization of the initially trained front-end encoder and back-end diffusion model, improving their joint loss function L. total Minimize; repeat the reverse iteration until the joint loss function L is reached. total The training converges or reaches the preset number of training epochs, resulting in a well-trained front-end encoder and back-end diffusion model.

[0201] The front-end encoder processing module is used to extract the content features and fundamental frequency features of the source audio and the target timbre of the target audio, embed them into the trained front-end encoder for preliminary conversion, and generate a preliminary conversion spectrum.

[0202] The back-end diffusion model processing module is used to embed the initial converted spectrum and the target timbre into the input trained back-end diffusion model for dual-path inverse iterative noise reduction processing, and output the spectral components corresponding to the two paths.

[0203] The synthesis and conversion module is used to add the spectral components output by the back-end diffusion model processing module element by element and convert them into audio waveforms to complete the timbre conversion processing of the singing voice.

[0204] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the zero-sample singing timbre conversion method described above.

[0205] In this embodiment, the computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. The computer device includes, but is not limited to, devices that can communicate with each other via a system bus, such as memory, processor, and network interface.

[0206] A memory includes at least one type of computer-readable storage medium, which can be volatile or non-volatile. Specifically, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage module of a computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory can also include both internal storage modules and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device, such as program code for a voice timbre conversion method. In addition, memory can also be used to temporarily store various types of data that have been output or will be output.

[0207] A processor, in some embodiments, may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other voice-to-tone conversion chip. This processor is typically used to control the overall operation of a computer device, such as performing control and processing related to data interaction or communication with the computer device. In this embodiment, the processor is used to run program code stored in memory or process data.

[0208] A network interface, which may include a wireless network interface or a wired network interface, is typically used to establish communication links between computer devices and other computer devices. For example, a network interface is used to connect a computer device to an external terminal via a network, establishing data transmission channels and communication links between the computer device and the external terminal. Networks can be intranets, the Internet, Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), 4G networks, 5G networks, Bluetooth, Wi-Fi, and other wireless or wired networks.

[0209] In this embodiment, the vocal timbre conversion method stored in the memory can be divided into one or more program modules and executed by one or more processors (in this embodiment, a processor) to complete this application.

[0210] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the vocal timbre conversion method in the embodiment.

[0211] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the singing voice timbre conversion method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0212] The system used in the application environment of this invention may include a terminal device and a server. The terminal device and the server are connected via a wireless or wired network. The timbre conversion method provided in the embodiments of this invention can be executed by the server, by the terminal device, or by a combination of both. Accordingly, all modules included in the timbre conversion system can be located entirely in the server, entirely in the terminal device, or separately in the server and the terminal device.

[0213] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across at least two network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of this application. Those skilled in the art can understand and implement this without any creative effort.

Claims

1. A zero-sample singing timbre conversion method, characterized in that, The method includes the following steps: S1. Extract content features, fundamental frequency features, and timbre embeddings from the audio dataset; S2. After constructing the front-end encoder and back-end diffusion model, initial training is performed. A joint training strategy is adopted, utilizing the content features, fundamental frequency features, and timbre embeddings of the dataset audio to synchronously optimize the initially trained front-end encoder and back-end diffusion model end-to-end, thereby improving their joint loss function L. total Minimize; repeat the reverse iteration until the joint loss function L is reached. tota The training converges or reaches the preset number of training epochs, resulting in a well-trained front-end encoder and back-end diffusion model. S3. Extract the content features and fundamental frequency features of the source audio and the target timbre of the target audio, embed them into the trained front-end encoder for preliminary conversion, and generate a preliminary conversion spectrum. S4. Embed the preliminary conversion spectrum and target timbre from step S3 into the pre-trained back-end diffusion model for dual-path inverse iterative noise reduction processing, and output the spectral components corresponding to the two paths. S5. The spectral components output in step S4 are added element by element and converted into an audio waveform to complete the timbre conversion process of the singing voice.

2. The zero-sample singing timbre conversion method according to claim 1, characterized in that, The front-end encoder in step S2 is initially trained using the following method: The content features, fundamental frequency features, and timbre embeddings of the dataset audio extracted in step S1 are used as initial inputs, and ordinary data batches matching the content features, fundamental frequency features, and timbre embeddings are loaded from them. Convert the ordinary data batch into a mixed batch where the content features, fundamental frequency features, and timbre embeddings do not match; Merge ordinary data batches and mixed batches to form a large batch set; The front-end encoder generates a predicted Mel spectrum based on this large batch set, and obtains the total loss function of the front-end encoder by weighting the Mel reconstruction loss function and harmonic loss function, and then calculates the total loss of the front-end encoder. Then, by adjusting the loss weight hyperparameters, the total loss of the front-end encoder is minimized using backpropagation, thus completing the initial training of the front-end encoder.

3. The zero-sample singing timbre conversion method according to claim 2, characterized in that, The preliminary training method for the front-end encoder in step S2 is as follows: A data batch containing N audio samples is randomly selected from the audio dataset. For each audio sample in the data batch, the corresponding content features, fundamental frequency features, and timbre embeddings are extracted using the method described in step S1, thereby forming a set of N data samples with mutually matched features, denoted as the ordinary set B. normal ; Define a random permutation function that operates on the index set {1, 2, ..., N}, and use the random permutation function to transform the ordinary set B. normal The data sample that is converted into a mixed batch containing N content features, fundamental frequency features, and mismatched timbre embeddings is called the mixture set B. mixed ; The ordinary set B normal With the mixed set B mixed Perform a union operation to form the final large batch set B used for training. large : B large =B normal ∪B mixed The large batch set B large It contains 2N data samples; Large batch set B large The input to the front-end encoder generates a predicted Mel spectrum for each sample; Select the corresponding ordinary set B from the generated predicted Mel spectrum. normal The portion is compared with the target Mel spectrum, and the Mel reconstruction loss function and harmonic loss function are calculated respectively. The total front-end loss function of the front-end encoder is then constructed, and the total front-end loss L of the front-end encoder is calculated. fe ; L fe =λ mel L mel +λ har L har Among them, L fe This represents the total loss at the front end; L mel Reconstruct the loss for Mel; L har For harmonic loss; λ mel and λ har These are the preset Mel reconstruction loss weight hyperparameters and harmonic loss weight hyperparameters, respectively. Adjust the preset hyperparameters of Mel reconstruction loss and harmonic loss weights, and use the backpropagation algorithm to make the gradient of the total loss flow back to the front-end encoder, so that the total loss of the front-end encoder reaches the minimum, the iteration terminates, and the initial training of the front-end encoder is completed.

4. The zero-sample singing timbre conversion method according to claim 3, characterized in that, The initial training method for the backend diffusion model is as follows: The preliminary source spectrum and preliminary filter spectrum of the front-end encoder output are randomly selected as complementary conditions and are input together with the timbre embedding into the dual-path score estimation network. The diffusion process generates perturbation data and calculates the true scores, which drive the dual-path score estimation network to estimate the scores separately. The network parameters are then optimized collaboratively by minimizing the score matching loss, thus completing the initial training of the backend diffusion model.

5. The zero-sample singing timbre conversion method according to claim 4, characterized in that, The preliminary training method for the backend diffusion model is as follows: Randomly select one set of complementary conditions from the two sets of outputs generated by the front-end encoder: if the conditions of the source fraction estimation network are selected from the preliminary source spectrum of the mixed set, then the conditions of the filter fraction estimation network are selected from the preliminary filter spectrum of the ordinary set, and vice versa, thus completing the construction of complementary settings. A time point t is randomly selected from the continuous time interval [0, T], and a standard Gaussian noise z is also selected. Let the target Mel spectrum M target Starting from the initial data x(0), the perturbation data x of the source path is generated using standard Gaussian noise z and the analytical solution of the forward SDE. src (t) and the perturbation data x of the filter path flt (t); Based on the perturbation data x from the source path src (t) and the perturbation data x of the filter path flt Given x(t) and the initial data x(0), calculate the true fractional function at that moment. and Where σ t It is a time-dependent noise scale; Source Score Estimation Network Dec src and filter fraction estimation network Dec flt Receive disturbance data x from the source path respectively src (t) and the perturbation data x of the filter path flt After embedding the timbre of the dataset audio (t), time t, and complementary settings, the estimated values ​​of the true scores at that time are estimated respectively. Through backpropagation, the score matching loss L is used. diff The resulting gradient iteratively updates the weights of the source score estimation network and the filter score estimation network until the score matching loss L... diff Minimum, followed by initial training of the downstream diffusion model.

6. The zero-sample vocal timbre conversion method according to any one of claims 1 to 5, characterized in that, Step S3 specifically involves: extracting the source content features C of the source audio. src Source fundamental frequency characteristics P src And the target audio's target timbre embedding spk tgt The input is fed into the trained front-end encoder Enc. FE In this process, the source encoder and filter encoder in the front-end encoder are used for parallel path processing. The source encoder receives the source fundamental frequency feature P. src embed the target tone into spk tgt Output the preliminary source spectrum after initial conversion. Filter encoder received source content characteristics C src embed the target tone into spk tgt Output the preliminary filter spectrum after initial conversion.

7. The zero-sample singing timbre conversion method according to claim 6, characterized in that, Step S4 specifically involves: Dec utilizes two parallel source score estimation networks in a back-end diffusion model. src and filter fraction estimation network Dec flt High-quality spectral components are generated from random noise through an iterative reverse sampling process. The specific method for generating high-quality spectral components from random noise through an iterative inverse sampling process is as follows: Based on the selected sampler algorithm, sample from the set number of inference steps N. infer Initially, the inverse sampling process is executed iteratively until time step 0. In each sampling step, the sampler calls the source score estimation network Dec. src It provides the noisy input of the current path, the current sampling step number, and the preliminary source spectrum as conditions. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step; simultaneously, the sampler calls the filter fractional estimation network Dec. flt It provides the noisy input of the current path, the current sampling step number, and the preliminary filter spectrum as conditions. embed the target tone into spk tgt This is used to calculate the spectral estimate for the next sampling step; after inference sampling steps N infer After step-by-step iterative sampling, the final source components x0 of the two paths are obtained respectively. src and final filter components 8. The zero-sample singing timbre conversion method according to claim 7, characterized in that, Specifically, S5 is: Dec src and filter fraction estimation network Dec flt The final source component of the output and final filter components Element-wise addition is performed to obtain a high-fidelity Mel spectrum, which is then input into a vocoder to convert it into a playable audio waveform, thus completing the timbre conversion process of the singing voice.

9. The zero-sample singing timbre conversion method according to claim 1, characterized in that, The specific method for extracting the content features, fundamental frequency features, and timbre embedding of the dataset audio in step S1 is as follows: S11. Load the audio dataset D and extract audio sample A from the audio dataset D. raw ; S12, For audio sample A raw Data augmentation is performed by applying random transformations of pitch and formants, and content features C are extracted from the augmented audio using a content encoder; S13. Using a pitch extraction algorithm to extract from audio sample A raw The fundamental frequency curves of each frame are obtained, and the fundamental frequency value f0(t) of each frame is converted into a more uniform scale in perception to obtain the fundamental frequency feature P. S14. Using the audio preprocessing module to process audio sample A raw Extract the target Mel spectrum M target ; S15, Using a timbre encoder to extract audio sample A raw Extract the timbre features used to represent the speaker's identity and embed them as timbre embeddings (spk).

10. A zero-sample singing timbre conversion system, characterized in that, include: The feature extraction module is used to extract content features, fundamental frequency features, and timbre embeddings from the audio data set. Model training module: Used for initial training after building the front-end encoder and back-end diffusion model sequentially. It employs a joint training strategy, utilizing the content features, fundamental frequency features, and timbre embeddings of the dataset's audio to perform end-to-end synchronous optimization of the initially trained front-end encoder and back-end diffusion model, improving their joint loss function L. total Minimize; repeat the reverse iteration until the joint loss function L is reached. total The training converges or reaches the preset number of training epochs, resulting in a well-trained front-end encoder and back-end diffusion model. The front-end encoder processing module is used to extract the content features and fundamental frequency features of the source audio and the target timbre of the target audio, embed them into the trained front-end encoder for preliminary conversion, and generate a preliminary conversion spectrum. The back-end diffusion model processing module is used to embed the initial converted spectrum and the target timbre into the input trained back-end diffusion model for dual-path inverse iterative noise reduction processing, and output the spectral components corresponding to the two paths. The synthesis and conversion module is used to add the spectral components output by the back-end diffusion model processing module element by element and convert them into audio waveforms to complete the timbre conversion processing of the singing voice.

Images