VITS-based singing voice conversion method

Abstract

The invention discloses a singing voice conversion method based on VITS. The method comprises the following steps of: 1, respectively extracting a bottleneck feature, a fundamental frequency, a singing style feature and a linear spectrogram from the singing of a source singer by utilizing a Whisper superficial layer encoder, an automatic singing transcription model AST and an AutoVC-based pitch-energy encoder, and taking the bottleneck feature, the fundamental frequency, the singing style feature and the linear spectrogram as the input of a VITS-based singing voice conversion model; reconstructing the singing voice by using a VITS-based singing voice conversion model in combination with the voiceprint characteristics of the singer to obtain converted singing voice of the target singer; and 2, performing pitch adjustment according to the pitch difference between the source singer and the target singer through a pitch shifter, and generating a target singing sound with the characteristics of the target singer by taking the bottleneck characteristics, the F0 and the singing style characteristics as the input of a voice conversion model. The method has the characteristics of multi-level refined acoustic feature decoupling, explicit singing style migration, high-fidelity waveform generation and dynamic adaptive pitch conversion.

Description

Technical Field

[0001] The invention belongs to the technical field of voice conversion, and in particular relates to a singing voice conversion method based on VITS. Background Art

[0002] In recent years, voice conversion technology has made significant progress in the field of speech synthesis. Its core goal is to convert acoustic features such as timbre and intonation to the characteristics of the target speaker while preserving the source speech content. However, singing voice conversion faces more complex challenges: singing voice conversion must not only accurately convey the lyrics, but also preserve artistic characteristics such as melody, rhythm, and emotional expression. It also needs to overcome issues such as a wide dynamic range of pitch, diverse singing styles, and highly personalized timbre and vocalization methods. Furthermore, due to the high human and financial costs associated with parallel data, singing voice conversion based on non-parallel data has gradually become a mainstream research direction.

[0003] Traditional singing voice conversion methods usually extract the fundamental frequency, spectral envelope and other features of the voice first, and then adjust them to the target features through a mapping function. However, this method has the limitations of information loss and error accumulation. Deep learning-based methods have become a research hotspot due to their high-quality waveform output. For example, UCD-SVC, based on a generative framework centered around GANs, ignores non-periodic components, resulting in distorted details in the converted sound. So-VITS-SVC, based on the variational inference with adversarial learning for end-to-end text-to-speech (VITS) framework, inherits the end-to-end nature of VITS. However, these models typically rely on implicit feature modeling, resulting in weak explicit control over singing style characteristics and a lack of similarity. Fast SVC, for example, outputs the target waveform through a single model similar to VITS or GAN-based models. However, to balance speed and quality, they simplify feature extraction and generator design. Their relatively coarse acoustic feature decoupling capabilities can easily lead to timbre confusion and lack robustness. DiffSVC, based on diffusion models, strives to generate high-quality waveforms, but still suffers from insufficiently refined style transfer.

[0004] In addition, the core challenges of singing voice conversion remain the difficulty in feature decoupling caused by the continuous pitch changes and complex harmonic structure of singing voice, and the difference between the target singer's pitch range and the source singer's pitch range may cause distortion in the generated voice. Summary of the Invention

[0005] To overcome the shortcomings of the aforementioned prior art, the present invention aims to provide a singing voice conversion method based on VITS. The VITS framework can accept acoustic features of various dimensions, such as pitch, rhythm, and speech content, as input. This more accurate and detailed feature input facilitates complex and natural singing voice conversion. This singing voice conversion method uses F0, bottleneck features, voiceprint features, and singing style features based on pitch and energy profiles as input. By integrating variational inference and adversarial training, it achieves high-quality, highly natural singing voice conversion. It features multi-level, refined acoustic feature decoupling, explicit singing style transfer, high-fidelity waveform generation, and dynamic, adaptive pitch conversion.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A singing voice conversion method based on VITS comprises the following steps:

[0008] Step 1: Training phase:

[0009] Using Whisper shallow encoder, automatic singing transcription model (AST), and AutoVC-based pitch-energy encoder, bottleneck features (BNF), fundamental frequency (F0), singing style features (Style), and linear spectrogram (Linear Spectrogram) are extracted from the singing voice, respectively. These are used as input for the VITS-based singing voice conversion model.

[0010] The target singer's voiceprint features (VPF) are used to reconstruct the singing voice using the VITS-based singing voice conversion model to obtain the converted singing voice.

[0011] Step 2: Reasoning stage:

[0012] The pitch shifter is used to adjust the pitch according to the pitch difference between the source singer and the target singer. The bottleneck feature, fundamental frequency, and singing style feature are used as the input of the singing voice conversion model to generate a target singing voice with the characteristics of the target singer.

[0013] In step 1, the Whisper shallow encoder is used to extract bottleneck features (BNF), capturing voice content features unrelated to the singer while suppressing source timbre interference; the automatic singing transcription model AST is used to extract fundamental frequency (F0) and capture the continuous pitch changes of singing voice; the AutoVC-based pitch-energy encoder is used to extract explicit singing style features and explicitly model the singing style.

[0014] In step 1, the process of extracting bottleneck features from the shallow encoder layer of the Whisper model is as follows:

[0015] (1) Preprocessing

[0016] The spectrogram obtained by inputting the original audio is normalized so that its value range is between [-1, 1] and the mean of the input data is ensured to be close to 0; the purpose is to convert the audio signal into a feature representation that can be processed by the Whisper model.

[0017] (2) Process audio features through the Transformer encoder.

[0018] The Whisper model uses a Transformer-based encoder to process audio features. After initial processing through a series of convolutional layers, the audio features are input into the Transformer encoder, which then processes them through multiple layers of self-attention and feedforward networks to generate a high-level representation of the audio. The output of each layer contains increasingly abstract audio information.

[0019] (3) Extract the bottleneck features of the 20th layer

[0020] The bottleneck features of the Whisper model are extracted from the middle and shallow layers of the Transformer encoder, especially from the 20th layer. The 20th layer is a shallow encoder layer, and its output represents the high-level semantic information of the audio signal, which is a key node for the model to understand the audio content.

[0021] In step 1, voiceprint feature extraction uses Mel-Spectrogram, and the extraction process is as follows:

[0022] The time domain signal x(t) is directly obtained from the time domain waveform of the input original audio. The time domain signal x(t) is divided into frames through STFT and the spectrum X(m,k) of each frame is calculated, as shown in formula (1):

[0023]

[0024] Where m represents the frame index, k represents the frequency index, w(n) represents the window function, using Hamming window, and N represents the number of FFT points.

[0025] Then square the spectrum amplitude to obtain the power spectrum P(m,k) as shown in formula (2):

[0026] P(m,k)=|X(m,k)| 2 (2)

[0027] The Mel frequency scale describes the nonlinear characteristics of the human ear, that is, the low-frequency resolution is higher and the high-frequency resolution is lower. Its calculation method is shown in formula (3):

[0028]

[0029] Where f represents the linear frequency, f mel Indicates the corresponding Mel frequency;

[0030] On the Mel frequency scale, the Mel filter uses a triangular window function to weight the power spectrum, and the filter response is calculated by the triangular function, as shown in formula (4):

[0031]

[0032] where f i+1 , f i+2 Respectively represent the boundary frequencies of the two adjacent filters of the i filter;

[0033] Triangular filter bank H i (k) The formula for weighting the power spectrum P(m,k) is shown in formula (5):

[0034]

[0035] Among them, S(m,i) represents the output of the mth frame on the i-th filter;

[0036] In order to make the signal more consistent with human ear perception, the dynamic range of the weighted power spectrum is compressed using logarithmic transformation, as shown in formula (6):

[0037] Log-Mel(m,i)=log(S(m,i)+∈) (6)

[0038] Among them, ∈ is a smoothing constant to prevent zero values;

[0039] Finally, the features of each frame are concatenated into a two-dimensional matrix to form a Mel-Spectrogram.

[0040] Furthermore, in order to eliminate the influence of channel differences and noise on features, standardize the feature distribution, and improve robustness, the filter output needs to be normalized, as shown in formula (7):

[0041]

[0042] Among them, F(m,k) represents the value of Mel-Spectrogram in the mth frame and the kth dimension, μ k represents the mean of the feature in the kth dimension, σ k Represents the standard deviation of the feature in the kth dimension;

[0043] PCA is used to reduce the data dimension and extract the characteristic component matrix of each Mel-Spectrogram group, which consists of n components, each component has d variables, and the sample matrix X is shown in formula (8):

[0044]

[0045] In order to eliminate the influence of the magnitude of the eigenvalue, the mean and variance of each column are calculated and standardized, as shown in Equations (9), (10), and (11):

[0046]

[0047] in, represents the mean of the kth column of the feature matrix, represents the variance of the kth column of the feature matrix;

[0048] Then calculate the correlation coefficient matrix and eigenvalue λ of the standardized features i , and sort them from large to small. According to the principle that the principal component contribution rate is greater than 90%, calculate the cumulative contribution rate of the first m principal components. Construct a dimensionality reduction matrix, as shown in formula (12):

[0049]

[0050] The reduced feature matrix is used as the input of BiLSTM. The forward and backward LSTMs process the sequence simultaneously, capturing the dependencies between the features of each frame. The hidden state of each frame is calculated in sequence. The output of each frame is the concatenation of the forward and backward hidden states. The final output is used as the input of the subsequent network.

[0051] The update formulas of Mel-Spectrogram features in LSTM units after dimensionality reduction are shown in equations (13), (14), (15), (16), and (17):

[0052] f t =σ(W f ·[h t-1 ,x t ]+b f ) (13)

[0053] i t =σ(W i ·[h t-1 ,x t ]+b i ) (14)

[0054]

[0055] h t =o t ⊙tanh(C t ) (17)

[0056] Among them, f t 、it 、o t Represent the forget gate, input gate and output gate in the LSTM unit respectively, C t Represents a memory unit, h t represents the hidden state, σ represents the sigmoid function, W f 、W i 、W c represents the weight matrix, b f 、b i 、b c represents the bias term in the forget gate, input gate, and output gate, h t-1 represents the hidden state of the previous time step, x t Represents the input features of the current time step, ⊙ represents the element-by-element matrix multiplication;

[0057] The forward and backward calculations of the bidirectional LSTM are shown in Equations (18), (19), and (20):

[0058]

[0059] Among them, h t Represents the concatenation of the forward and backward hidden states.

[0060] The explicit singing style feature extraction is achieved through the singing style conversion encoder:

[0061] The encoder consists of two parts: pitch contour conversion and energy contour conversion;

[0062] The pitch conversion part is used to convert the pitch contour of the input singer into the pitch style of the target singer, and the energy conversion part adjusts the energy contour to match the style of the target singer based on the pitch conversion.

[0063] The specific implementation process of the encoder is as follows:

[0064] Step 1: Input preprocessing: Use the automatic singing transcription model AST to extract the accurate initial pitch contour p, and use this parameter to calculate the logarithmic scale energy value of each frame of audio signal to obtain the initial energy contour e. The calculation method is shown in formula (21):

[0065]

[0066] Among them, x n Represents the sampling point value in the audio frame, N represents the number of sampling points in the frame, and e(t) represents the energy value at time t;

[0067] Convert p to MIDI embed p e , for non-integer MIDI values, linear interpolation is used to calculate the embedding, converting e into the embedding e e, and is evenly divided into intervals by logarithm;

[0068] To ensure the embedded pitch p e and energy e e To be restored, it is necessary to calculate the reconstruction loss L recon , as shown in formula (22):

[0069]

[0070] Among them, BCE stands for cross entropy loss, RMSE stands for root mean square loss, and weight λ b and λ r Controls the relative importance of BCE and RMSE.

[0071] In order to learn the singer's unique vibrato style and make the generated results more realistic and personal, it is necessary to optimize the model's learning and reconstruction capabilities of vibrato characteristics and calculate the vibrato loss L vibr , as shown in formula (23):

[0072]

[0073] Among them, p ft represents the spectral features extracted from the true pitch contour by short-time Fourier transform (STFT), represents the spectral features of the generated predicted pitch contour, p vaf vibrato amplitude characteristics that represent the true pitch contour, represents the vibrato amplitude feature for predicting pitch contour, and the weight λ ft and λ vaf Learn and restore the target singer's vibrato characteristics;

[0074] Step 2: Pitch and energy model encoder; extract the input pitch or energy embedding into a compact high-dimensional feature and limit the inflow of unnecessary information through the information bottleneck. The encoders of the two parts of the model are consistent, that is, the pitch embedding p e and energy embedded in e e 1D convolution is used to extract local temporal features. Four residual blocks, each containing two layers of 1D convolution, normalization, and ReLU activation functions, are then used to extract temporal information through a bidirectional LSTM, with an output dimension of 2. Finally, downsampling is performed to form an information bottleneck.

[0075] Step 3: Pitch and Energy Model Decoders: Both the pitch and energy decoders need to accept the output features of their respective encoders and the target singer embedding obtained by the lookup table (LUT). They are concatenated after upsampling to the same number of original frames and processed using the same 1D convolution, residual block, and bidirectional LSTM. The difference is that the energy decoder requires an additional input pitch contour during the decoding stage. After being spliced with other features, subsequent processing is performed; this cascade structure can use pitch information to guide energy style conversion and improve consistency.

[0076] Step 4: Post-processing: convert the embedded matrix output by the pitch contour model decoder The MIDI embedding value of each frame is mapped back to the actual predicted pitch by weighted averaging, and the pitch contour p and the predicted pitch contour are Apply sinc function to perform high frequency filtering to extract vibrato characteristics p sharp , as shown in formula (24):

[0077] p sharp =p-sinc(i)*p (24)

[0078] Where * represents convolution.

[0079] Then extract the spectrum loss through STFT As shown in formula (25), ensure that the frequency distribution conforms to the target singer's style:

[0080]

[0081] The amplitude feature loss is obtained by comparing the root mean square error between the true and predicted amplitude profiles. As shown in formula (26), the vibrato strength characteristics of the target singer are captured.

[0082]

[0083] The smoothness loss is obtained by measuring the smoothness of the amplitude change using the first-order difference As shown in formula (27), the artificial sense caused by sudden amplitude changes is avoided;

[0084]

[0085] Finally, the total vibration loss is integrated As shown in Equation (28), the pitch contour generated by the model is guaranteed to be and energy profile Match the target style;

[0086]

[0087] The weight parameter λ ft =λ vaf =λ smooth =0.1;

[0088] Decoder output It is the final result after transformation;

[0089] This completes the transformation of performance style and obtains the pitch contour that highlights the performance style. and energy profile

[0090] The singing voice conversion model includes four parts: a priori encoder, a posteriori encoder, a decoder and a discriminator;

[0091] The posterior encoder and decoder work together through a self-reconstruction mechanism to simulate the hidden representation z of the singing waveform y and predict the original singing waveform. The discriminator performs adversarial training on the generated singing waveform to ensure the quality of waveform generation. The prior encoder captures the pitch changes and language content characteristics in the source singing and integrates them with the timbre of the target singer. The prior encoder and posterior encoder are connected using a reversible normalization stream.

[0092] Specifically:

[0093] (1) A priori Encoder

[0094] Using a multi-layer Transformer, the input feature c is transformed using its self-attention mechanism and multi-head attention mechanism. bnf 、c f0 、c style and c vpf Efficient fusion is performed at multiple levels, and then the prior distribution p(z|c bnf ,c f0 ,c style ,c vpf ), and finally outputs the latent variable z, which is used together with the subsequent decoder to generate the final singing waveform;

[0095] Bottleneck features extracted by Whisper model (using c bnf ), F0 features extracted by the automatic singing transcription model (denoted by c f0 denoted by), singing style features extracted based on AutoVC (expressed by c style denoted) and voiceprint features (denoted by c vpf express);

[0096] (2) Posterior Encoder

[0097] The non-causal WaveNet structure is used for construction. Its specific task is to use the linear spectrogram generated by the singing waveform y as input, use the residual block in the WaveNet structure to learn features at multiple time steps, capture the local dependencies of the input singing, and output the latent variable z through the processing of the convolution layer. The variable follows the posterior distribution p(z|y);

[0098] During the training process, the output of the prior encoder and the posterior encoder are subjected to KL divergence loss calculation to narrow the gap between the prior distribution and the posterior distribution. Minimizing the KL divergence loss helps to make the output of the prior encoder consistent with that of the posterior encoder.

[0099] (3) Decoder

[0100] The decoder's task is to generate the target singing waveform based on the hidden representation z The decoder part of the VITS architecture is built based on HIFI-GAN (a generative adversarial network for high-fidelity speech synthesis) and uses a solution combined with a neural source filter to implement the decoding process;

[0101] The improved decoder mainly consists of a multi-receptive field fusion (MRF) module, a source module, and a filter module;

[0102] The multi-receptive field fusion module is derived from HIFI-GAN, and the source module converts F0 into an excitation signal e by combining a sine wave. t , which is defined as shown in formula (29). When the frequency is not zero, that is, in the sound stage, the excitation signal e t It is composed of F0 and harmonics. In the silent stage, the excitation signal e t is composed of Gaussian noise;

[0103]

[0104] Among them, α represents the amplitude coefficient, which is set to 0.1, n t represents Gaussian random noise, σ represents the standard deviation of Gaussian random noise, which is set to 0.003, and n t ~N(0,0.003 2 ), N s represents the waveform sampling rate, φ∈[-π,π] represents the random initial phase;

[0105] In the filter module, the non-AR dilated convolution network generates the excitation signal e t After being fused with the latent representation z, a high-pass filter processes the silent part, or the noise part, to preserve the high-frequency components; a low-pass filter processes the vocal part, or the harmonic part, to preserve the low-frequency components. Through a more granular audio modeling approach, pitch is precisely controlled, naturally generating singing with coherent pitch changes. The voice flag is used to indicate whether the current audio is voiced or silent.

[0106] (4) Discriminator

[0107] The discriminator uses a combination of multi-period discriminator (MPD) and multi-scale discriminator (MSD). In the singing voice conversion model, the training of the discriminator is adversarial, and the generator part is played by the decoder to generate the target singing waveform close to the real singing voice. The task of the discriminator is to judge whether the generated singing waveform is natural and consistent with the timbre of the target singer, to ensure that the generated singing is as close to the target singer as possible, to improve the quality of the generated model, and to maximize the naturalness and realism of the audio.

[0108] In the multi-period discriminator, the input audio waveform is divided into subsequences according to different periods and input into multiple discriminator branches MPD respectively. i , each branch outputs two types of information: the probability value P of whether the input is real audio i and the intermediate layer features F for feature matching i , then assign weights to the losses of each cycle branch, and sum them up as the total loss L MPD , as shown in formula (30):

[0109]

[0110] Among them, L adv,i represents the independent adversarial loss of each cycle branch, w i Indicates the allocation weight.

[0111] The multi-scale discriminator analyzes different frequency bands and evaluates the low-frequency and high-frequency parts of the audio separately. The multi-scale discriminant method helps the generator adjust the details of the generated audio and optimize it in different frequency bands, which can effectively avoid the spectral distortion problem of the generated audio.

[0112] In the multi-scale discriminator, the input audio is converted into branches of different resolutions by downsampling at different times, and input into multiple discriminator branches MSD respectively. i , each scale branch outputs a probability value P i and the intermediate layer features F i , then assign weights to the losses of each scale branch, and sum them up as the total loss L MSD , as shown in formula (31):

[0113]

[0114] Among them, L adv,i represents the independent adversarial loss for each scale branch.

[0115] Each multi-cycle branch and multi-scale branch intermediate feature map (such as each layer of convolution output) participates in the feature matching loss L FM The calculation of is shown in formula (32):

[0116]

[0117] in, represents the l-th layer feature map of the i-th discriminator, C i and H i represents the number of channels and feature map length, ‖*‖1 represents the L1 norm;

[0118] Final discriminator loss L total It is the weighted sum of MPD loss, MSD loss and feature matching loss, as shown in Equation (33):

[0119]

[0120] The combined use of multi-period discriminators and multi-scale discriminators not only helps the generator optimize the global features of the audio (such as tone and pitch), but also helps the generator optimize the spectral details and timbre at the micro level. In the adversarial training between the generator and the discriminator, the final generated singing voice is more realistic and natural.

[0121] The working principle of the pitch shifter in step 2 is divided into the following steps:

[0122] Step 1: Determine the pitch P of the source singer in a given song or sentence s , the target singer's pitch P t , and calculate the deviation ΔP between the source pitch and the basic pitch, as shown in formula (34):

[0123] ΔP=P s -P t (34)

[0124] Step 2: Calculate the source singer's pitch variance and the target singer's pitch variance As shown in Equations (35) and (36), the variance reflects the degree of pitch variation within the pitch range of each singer. Considering the variance can make the adjustment process smoother and adapt to the characteristics of different pitch distributions;

[0125]

[0126] Among them, P s,i and P t,i Represent the pitch samples of the source singer and the target singer respectively, μ s and μ t Represents the mean of the source singer and the target singer respectively, N s and N t Represents the number of samples of source singer and target singer respectively;

[0127] Step 3: Adjust the source singer's pitch based on the difference and variance between the source singer and the target singer to make it closer to the target singer's pitch. The adjusted pitch P p The calculation method of is shown in formula (37):

[0128]

[0129] α controls the adjustment range of the pitch difference, and the range is set to 0.5≤α≤1.5. If α is large, the source pitch will be more offset to the target pitch. β controls the adjustment range of the variance difference, and the range is set to 0.1≤β≤0.5. If β is large, the source singer's pitch range will be closer to the target singer's pitch range, avoiding the impact of too small or too large variance.

[0130] Step 4: Adjusted Pitch P p It is passed to the singing voice conversion model to provide fundamental frequency parameters that match the target singer's vocal range; the target timbre bound to the language content and rhythm provided by the bottleneck feature, the dynamic singing style characteristics injected by the singing style feature, and the voiceprint feature is fused through the multi-layer Transformer prior encoder of the singing voice conversion model, and then the decoder generates the final singing waveform of the target singer for a given song. The adjustment of the pitch shifter ensures that the generated target singing voice is consistent with the target singer's performance in pitch, thereby improving the naturalness and similarity of the conversion result.

[0131] Beneficial effects of the present invention:

[0132] The present invention adopts Whisper shallow encoder to extract bottleneck features, suppressing source timbre interference while retaining language content and prosodic information; constructs a bidirectional LSTM voiceprint encoder based on Mel-Spectrogram, combines PCA dimensionality reduction and dynamic time series modeling, and improves the discriminability of the target singer's timbre characteristics.

[0133] The present invention designs a singing style conversion encoder based on pitch-energy cascade, which explicitly models the dynamic style characteristics of the target singer and enhances the style transfer capability of the model.

[0134] An improved singing voice conversion model architecture based on VITS. The decoder integrates a neural source filter, combining a sinusoidal excitation signal with neural network filtering to enhance the coherence of harmonic generation. The discriminator uses a multi-period and multi-scale joint optimization strategy to balance the authenticity of pitch periodicity and spectral details.

[0135] The present invention introduces a variance-driven pitch shifter to dynamically adjust the pitch distribution to adapt to the natural vocal range of the target singer and avoid pitch distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0136] Figure 1 Schematic diagram of the VITS system of the present invention.

[0137] Figure 2 Schematic diagram of the singing voice conversion algorithm framework based on VITS.

[0138] Figure 3 Schematic diagram of PPC and SED indicators for training different speech conversion systems using different BNF layers.

[0139] Figure 4 Generate a flow chart for Mel-Spectrogram.

[0140] Figure 5 A diagram comparing the training accuracy of different voiceprint feature extractions.

[0141] Figure 6 Schematic diagram of the singing style transfer encoder.

[0142] Figure 7 Schematic diagram of the architecture of the VITS-based singing voice converter.

[0143] Figure 8 Schematic diagram of the improved decoder structure.

[0144] Figure 9 Schematic diagram of the discriminator workflow.

[0145] Figure 10 Schematic diagram for visualizing ablation experiment results.

[0146] Figure 11 Schematic diagram for visualizing the experimental results. DETAILED DESCRIPTION

[0147] The present invention will be described in further detail below with reference to the accompanying drawings.

[0148] like Figure 1 As shown in the figure, VITS is an end-to-end text-to-speech (TTS) synthesis system based on deep learning. Unlike the two-stage TTS system that requires the generation of intermediate speech representations such as mel-spectrograms during the process, VITS provides an end-to-end generation framework that generates speech waveforms directly from text. It combines conditional variational autoencoders, normalized flows, and adversarial training to make the generated speech closer to real speech and have rich rhythm and intonation variations.

[0149] The core modules of the VITS model architecture consist of a posterior encoder, a priori encoder, a decoder, a random duration predictor, and a discriminator. The working process of the VITS architecture is explained as follows, using the training process as an example:

[0150] Step 1: Input processing and alignment estimation. Convert the input text into a phoneme sequence c text And estimate the input phoneme sequence and the target speech x by Monotonic Alignment Search (MAS) mel The alignment matrix A is obtained by maximizing the log-likelihood of the latent variables, as shown in Equations (38) and (39):

[0151]

[0152] in, represents the candidate alignment matrix, represents the conditional probability of the latent variable z, f θ (z) represents the latent variable z transformed by the normalized flow, represents the mean of the conditional prior distribution, represents the standard deviation of the conditional prior distribution, is modeled as a normally distributed N.

[0153] Step 2: Prior and posterior encoding. The phoneme sequence is passed through a Transformer-based text encoder to generate a hidden representation h text Combined with the alignment matrix A, the prior distribution parameters are generated by linear projection, as shown in formula (40):

[0154] p θ (z|c text ,A)=N(f θ (z); μ θ (c text ,A),σ θ (c text ,A)) (40)

[0155] Then the target speech waveform y is converted into a linear spectrogram x lin , is used as the input of the posterior encoder, and the linear spectrogram features are extracted using the non-causal WaveNet residual block to generate the posterior distribution parameters, as shown in Equation (41):

[0156] q φ (z|x lin )=N(z;μ φ (x lin ),σ φ (x lin )) (41)

[0157] Step 3: Latent variable sampling and encoding. Sample latent variables z~q from the posterior distribution φ (z|x lin ), and then use the decoder to decode the latent variable z into a predicted waveform The predicted waveform is further transformed by short-time Fourier transform (STFT) Convert to predicted mel spectrogram

[0158] Step 4: Loss calculation. By calculating the predicted Mel spectrogram and the target Mel spectrogram x mel The reconstruction loss L is obtained by taking the element-wise absolute difference between recon ; measure the difference between the prior distribution and the posterior distribution to obtain the KL divergence loss L kl Calculate the phoneme duration d based on the alignment matrix, use the random duration predictor to model the distribution of phoneme duration, and obtain the duration prediction loss L by maximizing the log-likelihood training dur ; The adversarial loss is mainly composed of the discriminator loss, the discriminator loss L adv (D) As shown in formula (42):

[0159]

[0160] in It expresses the expectation of the joint distribution of random variables y and z.

[0161] Generator loss L adv (G) is shown in formula (43):

[0162]

[0163] Feature matching loss L fm (G) is shown in formula (44):

[0164]

[0165] Where ‖*‖1 represents the absolute value norm.

[0166] Integrating all losses can get the total loss function L vae , as shown in formula (45):

[0167] L vae =L recon +L kl +L dur +L adv (G)+L fm (G) (45)

[0168] Step 5: Optimize and update. Use the AdamW optimizer to perform gradient updates on all model parameters to minimize the total loss L vaeThe hyperparameters β1 and β2, as well as the learning rate, are set. To reduce computational overhead, a windowed generation strategy is used, generating only partial waveform segments rather than the entire speech. The model continuously optimizes the learning text-to-speech mapping relationship to generate high-quality, natural speech.

[0169] Singing voice conversion algorithm based on VITS:

[0170] Singing voice conversion algorithm framework:

[0171] Based on the VITS framework's end-to-end waveform-level modeling capabilities, it can effectively separate language content, timbre, and pitch information, and has the ability to generate target singer voice characteristics through normalized streams. This paper proposes a singing voice conversion method based on VITS. Figure 2 As shown, it includes two stages: training and inference; the training stage utilizes Whisper shallow encoder, automatic singing transcription model AST and pitch-energy encoder based on AutoVC;

[0172] Bottleneck features (BNF), F0, singing style features (Style) and linear spectrograms (Linear Spectrogram) are extracted from the singing voice respectively and used as inputs of the VITS-based singing voice conversion model. The singing voice conversion model mainly consists of a priori encoder, posterior encoder, decoder and discriminator, and the singing voice is reconstructed using the singer's voiceprint features (VPF).

[0173] In addition to feature extraction and singing voice conversion, the inference phase also incorporates a pitch shifter. Different singers have different pitch ranges, and the pitch shifter adjusts the pitch based on the pitch difference between the source and target singers. The speech conversion model uses bottleneck features, F0, and singing style features as input to generate a target singing voice with the characteristics of the target singer.

[0174] Bottleneck features and voiceprint feature extraction:

[0175] (1) Bottleneck feature extraction:

[0176] In singing voice conversion, accurate language content extraction is essential for generating intelligible speech. Extracting appropriate bottleneck features preserves some language content information, such as intonation and rhythm, while removing timbre-related features from the input audio. This prevents the source singer's timbre from leaking into the target singer's converted speech, thereby ensuring the quality of the singing voice conversion.

[0177] Bottleneck features are extracted from the shallow encoder layers of the Whisper model. In Whisper's shallow and mid-layers (layers 10-30), layers that are too shallow may only contain low-level acoustic information, which will not accurately represent language content and rhythm, while layers that are too deep may contain too much semantic information, leading to interference with the source timbre. Experimental verification has found that the 20th layer in the shallow layers is the optimal level, balancing the capture of language content and style information while avoiding timbre leakage, ensuring high fidelity and style modeling capabilities.

[0178] To validate the bottleneck features obtained from the 20th layer, the Pearson Correlation Coefficient (PCC) between the predicted waveform and the corresponding input speech is used to measure the content consistency between the speech conversion output and the input speech. A larger PPC indicates that more linguistic content and prosodic information of the input waveform is preserved. The Speaker Embedding Distance (SED) extracted from the pre-trained ECAPA-TDNN model is used to evaluate the timbre similarity between the speech conversion output and the target speech. A larger SED indicates a greater timbre similarity between the speech conversion output and the target speech.

[0179] Figure 3 The results of speech conversion trained using different BNF layers are shown. BNFs+Rec* represents a non-streaming BNF-based speech conversion system, while BNFs+Rec represents a streaming BNF-based speech conversion system. The figure shows that the SED of both the BNFs+Rec* and BNFs+Rec systems reaches its lowest value at the 20th layer, indicating that the 20th-layer BNF contains the timbre with the lowest similarity. From layers 10 to 30, the PPC of both systems is approximately decreasing, and the rate of decrease increases after the 20th layer, indicating that the shallow encoder layers contain richer linguistic content. To effectively capture linguistic content and some stylistic information while mitigating the influence of the source singer's timbre, extracting bottleneck features in the 20th encoder layer is optimal.

[0180] (2) Voiceprint feature extraction:

[0181] Voiceprint features are the core parameters used to characterize the target singer's timbre. They function like a "voiceprint fingerprint," ensuring that the converted voice is highly consistent with the target singer in terms of timbre, pronunciation style, and other aspects. Based on MFCC, PCA, and LSTM, this voiceprint feature extraction model combines classic signal processing methods with deep learning. MFCC features have good acoustic discriminability, but they can cause the loss of high-frequency and some detailed information during discrete cosine compression. High-frequency features are essential for voiceprint recognition. Therefore, to preserve more time-frequency features and enhance the ability to describe detailed features, especially high-frequency features, more richly, this paper uses Mel-Spectrogram instead of MFCC.

[0182] like Figure 4 The figure shows the flow chart for generating Mel-Spectrogram. The specific calculation process is as follows:

[0183] First, the time domain signal x(t) is divided into frames by STFT and the spectrum X(m,k) of each frame is calculated, as shown in formula (1):

[0184]

[0185] Wherein, m represents the frame index, k represents the frequency index, w(n) represents the window function, using Hamming window, and N represents the number of FFT points.

[0186] Then square the spectrum amplitude to obtain the power spectrum P(m,k) as shown in formula (2):

[0187] P(m,k)=|X(m,k)| 2 (2)

[0188] The Mel frequency scale describes the nonlinear characteristics of the human ear, that is, the low-frequency resolution is higher and the high-frequency resolution is lower. Its calculation method is shown in formula (3):

[0189]

[0190] Where f represents the linear frequency, f mel Indicates the corresponding Mel frequency.

[0191] On the Mel frequency scale, the Mel filter uses a triangular window function to weight the power spectrum. The filter response can be calculated using the triangular function, as shown in Equation (4):

[0192]

[0193] where f i+1 , f i+2 They represent the boundary frequencies of the two adjacent filters of the i filter.

[0194] Triangular filter bank H i (k) The formula for weighting the power spectrum P(m,k) is shown in formula (5):

[0195]

[0196] Among them, S(m,i) represents the output of the mth frame on the i-th filter;

[0197] In order to make the signal more consistent with human ear perception, the dynamic range of the weighted power spectrum is compressed using logarithmic transformation, as shown in formula (6):

[0198] Log-Mel(m,i)=log(S(m,i)+∈) (6)

[0199] Here, ∈ is a smoothing constant to prevent zero values from occurring.

[0200] Finally, the features of each frame are concatenated into a two-dimensional matrix to form a Mel-Spectrogram.

[0201] Furthermore, in order to eliminate the influence of channel differences and noise on features, standardize the feature distribution, and improve robustness, the filter output needs to be normalized, as shown in formula (7):

[0202]

[0203] Among them, F(m,k) represents the value of Mel-Spectrogram in the mth frame and the kth dimension, μ i represents the mean of the feature in the i-th dimension, σ i Represents the standard deviation of the feature in the i-th dimension.

[0204] PCA is a classic method for unsupervised learning and data dimensionality reduction. It transforms raw data into a new feature space and converts multiple metrics into several representative composite metrics that reflect most of the information in the original metrics while ensuring that each metric is independent of each other.

[0205] Using Mel-Spectrogram can generate dozens or even hundreds of features per frame, which can lead to excessive computational complexity for long speech signals. Furthermore, features of different dimensions can be strongly correlated, leading to data redundancy. To address these issues while improving feature discriminability and retaining features that contribute significantly to the data, PCA is used for data dimensionality reduction.

[0206] Extract each group of Mel-Spectrogram feature component matrix, which consists of n components, each component has d variables, and the sample matrix X is shown in formula (8):

[0207]

[0208] In order to eliminate the influence of the magnitude of the eigenvalue, the mean and variance of each column are calculated and standardized, as shown in Equations (9), (10), and (11):

[0209]

[0210] in, represents the mean of the kth column of the feature matrix, represents the variance of the kth column of the feature matrix.

[0211] Then calculate the correlation coefficient matrix and eigenvalue λ of the standardized features k , and sort them from large to small. According to the principle that the principal component contribution rate is greater than 90%, calculate the cumulative contribution rate of the first m principal components. Construct a dimensionality reduction matrix, as shown in formula (12):

[0212]

[0213] Voiceprint features are dynamic manifestations of speech signals, with strong temporal dependencies. BiLSTM can simultaneously capture forward and backward information to better understand the global temporal characteristics of speech and improve the discriminability of voiceprint features. The voiceprint extraction method uses the reduced feature matrix as input to the BiLSTM. The forward and backward LSTMs simultaneously process the sequence, capturing the dependencies between features within each frame. The hidden state of each frame is calculated sequentially. The output of each frame is the concatenation of the forward and backward hidden states, and the final output serves as the input to the subsequent network.

[0214] The update formulas of Mel-Spectrogram features in LSTM units after dimensionality reduction are shown in Equations (13), (14), (15), (16), and (17):

[0215] f t =σ(W f ·[h t-1 ,x t ]+b f ) (13)

[0216] i t =σ(W i ·[h t-1 ,x t ]+b i ) (14)

[0217]

[0218] h t =o t⊙tanh(C t ) (17)

[0219] Among them, f t 、i t 、o t Represent the forget gate, input gate and output gate in the LSTM unit respectively, C t Represents a memory unit, h t represents the hidden state, σ represents the sigmoid function, W f 、W i 、W c represents the weight matrix, b f 、b i 、b c represents the bias term in the forget gate, input gate, and output gate, h t-1 represents the hidden state of the previous time step, x t Represents the input features of the current time step, and ⊙ represents the element-by-element matrix multiplication.

[0220] The forward and backward calculations of the bidirectional LSTM are shown in Equations (18), (19), and (20):

[0221]

[0222] Among them, h t Represents the concatenation of the forward and backward hidden states.

[0223] For voiceprint feature extraction, 20 minutes of speech data from 10 female and 10 male speakers in the THCHS-30 dataset were used as training data. MFCC and Mel-Spectrogram features were extracted, subjected to PCA dimensionality reduction, and normalized before being fed into LSTM and BiLSTM networks for training. The training process was 100 rounds, with an initial learning rate of 0.1, a learning decay factor of 0.2, and a period of 20. The remaining training parameters were consistent with the network design parameters used by Xiong et al.

[0224] The experimental results are based on the accuracy rate. Figure 5 It can be seen that after about 600 iterations, all models can achieve an accuracy of more than 90%. After 1500 iterations, the accuracy gradually approaches 100%.

[0225] Analysis and comparison of the various models show that the BiLSTM model is generally superior to the LSTM model. The MFCC feature performs stably, but its accuracy is slightly lower than that of the Mel-Spectrogram feature. The combined model of Mel-Spectrogram+BiLSTM performs best, reaching an accuracy of 94% after 500 iterations. The accuracy increases steadily with the increase in the number of iterations. This proves that compared with the MFCC feature, the model using Mel-Spectrogram features combined with BiLSTM is more conducive to the extraction of voiceprint features.

[0226] Explicit singing style feature extraction:

[0227] In order to make the converted singing voice more natural and auditorily closer to the target singer's performance style, the present invention not only takes F0, bottleneck features and voiceprint features as input, but also explicitly models the singing style features contained in pitch and energy contours.

[0228] The present invention proposes a singing style conversion encoder, which consists of two parts: pitch contour conversion and energy contour conversion.

[0229] like Figure 6 As shown in the figure, the pitch conversion part is used to convert the pitch contour of the input singer into the pitch style of the target singer. The energy conversion part adjusts the energy contour based on the pitch conversion to match the style of the target singer. The specific implementation process of the encoder is as follows:

[0230] Step 1: Input preprocessing. Use the singing transcription model to extract the accurate initial pitch contour p, with a window size of 1024 and a frame shift size of 512. Use these parameters to calculate the logarithmic scale energy value of each frame of the audio signal to obtain the initial energy contour e. The calculation method is shown in Equation (21):

[0231]

[0232] Among them, x n Represents the sampling point value in the audio frame, N represents the number of sampling points in the frame, and e(t) represents the energy value at time t.

[0233] Convert p to a 144-dimensional MIDI embedding p e , linear interpolation is used to calculate the embedding for non-integer MIDI values, thereby ensuring pitch continuity and fine-grained features. Convert e to a 256-dimensional embedding e e , the energy value range is 10 -4 to 1, and is evenly divided into 256 intervals by logarithm.

[0234] To ensure the embedded pitch p e and energy e eTo be restored, it is necessary to calculate the reconstruction loss L recon , as shown in formula (22):

[0235]

[0236] Among them, BCE stands for cross entropy loss, RMSE stands for root mean square loss, and weight λ b and λ r Controls the relative importance of BCE and RMSE.

[0237] In order to learn the singer's unique vibrato style and make the generated results more realistic and personal, it is necessary to optimize the model's learning and reconstruction capabilities of vibrato characteristics and calculate the vibrato loss L vibr , as shown in formula (23):

[0238]

[0239] Among them, p ft represents the spectral features extracted from the true pitch contour by short-time Fourier transform (STFT), represents the spectral features of the generated predicted pitch contour, p vaf vibrato amplitude characteristics that represent the true pitch contour, represents the vibrato amplitude feature for predicting pitch contour, and the weight λ ft and λ vaf Learn and restore the target singer's vibrato characteristics.

[0240] Step 2: Pitch and Energy Model Encoder. The input pitch or energy embedding is extracted into a compact high-dimensional feature and the influx of unnecessary information is limited by the information bottleneck. The encoders of the two parts of the model are consistent, that is, the pitch embedding p e and energy embedded in e e Local temporal features are extracted through a layer of 1D convolution with a kernel size of 10. Then, four residual blocks, each consisting of two layers of 1D convolution, normalization, and ReLU activation functions, are used to extract temporal information through a bidirectional LSTM, with an output dimension of 2. Finally, downsampling is performed to form an information bottleneck.

[0241] Step 3: Pitch and Energy Model Decoders. Both the pitch and energy decoders need to accept the output features of their respective encoders and the target singer embedding obtained by the lookup table (LUT). They are concatenated after upsampling to the same number of original frames and processed using the same 1D convolution, residual block, and bidirectional LSTM. The difference is that the energy decoder requires an additional input of the pitch contour during the decoding stage. After being concatenated with other features for subsequent processing, this cascade structure can use pitch information to guide energy-style transfer and improve consistency.

[0242] Step 4: Post-processing. Convert the embedded matrix output by the pitch contour conversion model decoder to Map the MIDI embedding value of each frame back to the actual predicted pitch through weighted averaging Pitch contour p and predicted pitch contour Apply sinc function to perform high frequency filtering to extract vibrato characteristics p sharp , as shown in formula (24):

[0243] p sharp =p-sinc(i)*p (24)

[0244] Where * represents convolution.

[0245] Then extract the spectrum loss through STFT As shown in Equation (25), the frequency distribution is ensured to be consistent with the target singer’s style.

[0246]

[0247] The amplitude feature loss is obtained by comparing the root mean square error between the true and predicted amplitude profiles. As shown in formula (26), the vibrato strength characteristics of the target singer are captured.

[0248]

[0249] The smoothness loss is obtained by measuring the smoothness of the amplitude change using the first-order difference As shown in formula (27), the artificial sense caused by sudden amplitude changes is avoided.

[0250]

[0251] Finally, the total vibration loss is integrated As shown in Equation (28), the pitch contour generated by the model is guaranteed to be and energy profile Match the target style.

[0252]

[0253] The weight parameter λ ft =λ vaf =λ smooth =0.1.

[0254] Since energy is a continuously changing value, it does not need to be mapped back to a specific discrete note range. Moreover, through training, the energy decoder has learned the energy distribution characteristics of the target singer's style, including amplitude changes, dynamic range, etc., so the decoder output is It is the final result after conversion.

[0255] This completes the transformation of performance style and obtains the pitch contour that highlights the performance style. and energy profile

[0256] Singing Voice Conversion Model

[0257] Based on the VITS architecture, a singing voice conversion architecture is proposed, such as Figure 7 As shown in the figure, the architecture mainly consists of four parts: a priori encoder, a posterior encoder, a decoder, and a discriminator. The posterior encoder and decoder work together through a self-reconstruction mechanism to simulate the hidden representation z of the singing waveform y and predict the original singing waveform. The discriminator performs adversarial training on the generated singing waveform to ensure the quality of waveform generation. The prior encoder captures features such as pitch changes and language content in the source singing and integrates them with the timbre of the target singer. The prior encoder and posterior encoder are connected using a reversible normalization flow. The following is a detailed explanation of each part:

[0258] (1) A priori Encoder

[0259] In order to extract the bottleneck features (using c bnf ), F0 features extracted by the automatic singing transcription model (denoted by c f0 denoted by), singing style features extracted based on AutoVC (expressed by c style denoted) and voiceprint features (denoted by c vpf Representation) is effectively integrated, and a multi-layer Transformer and a reversible normalized flow structure are used to implement the prior encoder. Using a multi-layer Transformer, the input feature c is transformed into bnf 、c f0 、c style and c vpf Efficient fusion is performed at multiple levels, and then the prior distribution p(z|c bnf ,c f0 ,c style ,c vpf ), and finally outputs the latent variable z, which is used together with the subsequent decoder to generate the final singing waveform.

[0260] (2) Posterior Encoder

[0261] The posterior encoder inherits the design of the VITS architecture, employing a non-causal WaveNet structure. Its specific task is to take a linear spectrogram generated from the singing waveform y as input and leverage the residual blocks in the WaveNet structure to learn features over multiple time steps, capturing local dependencies within the input singing. The non-causal nature of the feature representation allows each time step to rely on both past and future information, thereby improving modeling capabilities. The convolutional layers output a latent variable z, which follows the posterior distribution p(z|y).

[0262] During training, the outputs of the prior and posterior encoders are subjected to a KL divergence loss to narrow the gap between the prior and posterior distributions. Minimizing the KL divergence loss helps align the outputs of the prior and posterior encoders, thereby improving the naturalness and stability of the singing voice conversion.

[0263] (3) Decoder

[0264] The decoder's task is to generate the target singing waveform based on the hidden representation z The decoder in the VITS architecture is built on HiFi-GAN. To reduce pitch errors and make the generated singing sound more natural and harmonious, an improved decoding method is proposed, incorporating a neural source filter. Unlike the original traditional decoder, which only uses a simple baseband conversion to generate the target singing voice, the improved decoder combines a sinusoidal excitation signal extracted from the baseband with a neural network filter, introducing a more complex and dynamic signal generation method, thereby more effectively simulating the audio characteristics of the singing voice.

[0265] like Figure 8 As shown in Figure 1, the improved decoder mainly consists of a multi-receptive field fusion (MRF) module, a source module, and a filter module. The multi-receptive field fusion module is derived from HIFI-GAN. The source module converts F0 into an excitation signal e by combining a sine wave. t , which is defined as shown in formula (29). When the frequency is not zero, that is, in the sound stage, the excitation signal e t It is composed of F0 and harmonics. In the silent stage, the excitation signal e t It is composed of Gaussian noise.

[0266]

[0267] Among them, α represents the amplitude coefficient, which is set to 0.1, n t represents Gaussian random noise, σ represents the standard deviation of Gaussian random noise, which is set to 0.003, and n t ~N(0,0.0032 ), N s represents the waveform sampling rate, and φ∈[-π,π] represents the random initial phase.

[0268] In the filter module, the non-AR dilated convolution network generates the excitation signal e t After being fused with the latent representation z, a high-pass filter processes the unvoiced (noise) portion of the signal to preserve high-frequency components. A low-pass filter processes the voiced (harmonic) portion of the signal to preserve low-frequency components. Through a more granular audio modeling approach, pitch is precisely controlled, naturally generating singing with coherent pitch variations. A voice flag indicates whether the current audio is voiced or silent.

[0269] (4) Discriminator

[0270] The discriminator adopts a combination of multi-period discriminator (MPD) and multi-scale discriminator (MSD). In the singing voice conversion model proposed in this invention, the training of the discriminator is adversarial, and the generator part is played by the decoder to generate the target singing waveform close to the real singing voice. The task of the discriminator is to determine whether the generated singing waveform is natural and consistent with the timbre of the target singer, ensuring that the generated singing voice is as close to the target singer as possible, improving the quality of the generated model, and maximizing the naturalness and realism of the audio.

[0271] Periodicity is crucial for speech, especially singing. Singing often contains rich melodies and pitch variations. These periodic characteristics, related to pitch and tone, determine whether the generated singing is harmonious and natural in pitch variation and melody. The multi-periodic discriminator primarily analyzes the periodicity of the audio signal, helping the generator adjust the fundamental frequency and pitch variations in the audio, enhancing the consistency and periodicity of the audio and avoiding the generation of unusual tones and unnatural pitches.

[0272] like Figure 9 As shown in (a), in the multi-period discriminator, the input audio waveform is divided into subsequences according to different periods and input into multiple discriminator branches MPD respectively. i , each branch outputs two types of information: the probability value P of whether the input is real audio i and the intermediate layer features F for feature matching i , then assign weights to the losses of each cycle branch, and sum them up as the total loss L MPD , as shown in formula (30):

[0273]

[0274] Among them, L adv,i represents the independent adversarial loss of each cycle branch, wi Indicates the allocation weight.

[0275] Tone and pitch are typically associated with low frequencies, while details like timbre and clarity are found in high frequencies. To ensure the overall quality of generated vocals, attention should be paid to each frequency band within the audio. The multi-scale discriminator analyzes different frequency bands, evaluating both the low and high frequencies of the audio. This multi-scale discriminant approach helps the generator fine-tune the details of the generated audio and optimize across different frequency bands, effectively avoiding spectral distortion in the generated audio.

[0276] like Figure 9 As shown in (b), in the multi-scale discriminator, the input audio is converted into branches of different resolutions by downsampling at different times, and input into multiple discriminator branches MSD respectively. i , each scale branch outputs a probability value P i and the intermediate layer features F i , then assign weights to the losses of each scale branch, and sum them up as the total loss L MSD , as shown in formula (31):

[0277]

[0278] Among them, L adv,i represents the independent adversarial loss for each scale branch.

[0279] Each multi-cycle branch and multi-scale branch intermediate feature map (such as each layer of convolution output) participates in the feature matching loss L FM The calculation of is shown in formula (32):

[0280]

[0281] in, represents the l-th layer feature map of the i-th discriminator, C i and H i represents the number of channels and feature map length, and ‖*‖1 represents the L1 norm.

[0282] Final discriminator loss L total It is the weighted sum of MPD loss, MSD loss and feature matching loss, as shown in Equation (33):

[0283] L total =L MPD +L MSD +λL FM (33)

[0284] The combined use of multi-period discriminators and multi-scale discriminators not only helps the generator optimize the global features of the audio (such as tone and pitch), but also helps the generator optimize the spectral details and timbre at the micro level. In the adversarial training between the generator and the discriminator, the final generated singing voice is more realistic and natural.

[0285] Pitch Shifter

[0286] Due to factors such as physiological conditions, vocal cord structure, and singing style, different singers have different pitch ranges, that is, the highest and lowest notes they can sing. Each singer performs most naturally and comfortably within their pitch range, which is also their optimal pitch range. If the source singer's pitch exceeds the target singer's natural pitch range, directly using the source singer's pitch to convert the target singer's voice may cause the target singer's voice to sound unnatural in certain pitch ranges, because the target singer may not be good at or accustomed to singing within these pitch ranges, resulting in the converted voice sounding unnatural and the similarity will be greatly reduced. And if the pitch does not match, the converted singing voice may have artifacts such as pitch confusion and pitch jumps, which will also affect the naturalness and listening experience of the voice.

[0287] To address the potential discrepancy in pitch range between the source and target singers, a pitch shifter is proposed. During inference, the source singer's pitch is adjusted to be closer to the target singer's pitch range. The working principle of the pitch shifter can be divided into the following steps:

[0288] Step 1: Determine the pitch P of the source singer in a given song or sentence s , the target singer's pitch P t , and calculate the deviation ΔP between the source pitch and the basic pitch, as shown in formula (34):

[0289] ΔP=P s -P t (34)

[0290] Step 2: Calculate the source singer's pitch variance and the target singer's pitch variance As shown in Equations (35) and (36), the variance reflects the degree of variation of pitch within the pitch range of each singer. Considering the variance can make the adjustment process smoother and adapt to the characteristics of different pitch distributions.

[0291]

[0292] Among them, P s,i and P t,i Represent the pitch samples of the source singer and the target singer respectively, μ s and μt Represents the mean of the source singer and the target singer respectively, N s and N t Represents the number of samples of the source singer and the target singer respectively.

[0293] Step 3: Adjust the source singer's pitch based on the difference and variance between the source singer and the target singer to make it closer to the target singer's pitch. The adjusted pitch P p The calculation method of is shown in formula (37):

[0294]

[0295] α controls the adjustment of the pitch difference, with a range of 0.5 ≤ α ≤ 1.5. A larger α will shift the source pitch closer to the target pitch. β controls the adjustment of the variance difference, with a range of 0.1 ≤ β ≤ 0.5. A larger β will bring the source singer's pitch range closer to the target singer's pitch range, avoiding the effects of too small or too large variance.

[0296] Step 4: Adjusted Pitch P p This information is passed to the singing voice conversion model, where it is combined with bottleneck features, singing style features, and voiceprint features to ultimately generate the target singer's singing voice for a given song. Adjustment of the pitch shifter ensures that the generated target singing voice is pitch-matched to the target singer's performance, improving the naturalness and similarity of the conversion result.

[0297] Experimental results analysis:

[0298] Training strategy and dataset:

[0299] The dataset used in the experiment is shown in the dataset used for training the singing voice conversion model. The training of the singing voice conversion model typically adopts a two-stage strategy consisting of pre-training and adaptation. To improve the model's adaptability to low-resource target singer data, a three-stage training strategy is introduced, consisting of a warm-up phase, a pre-training phase, and an adaptation phase, to optimize model performance. The specific training strategy is described as follows:

[0300] (1) Preheating stage:

[0301] The purpose of the warm-up phase is to use rich speech data to help the model initially learn acoustic features. This initializes the model's basic parameters and lays the foundation for subsequent singing data training. This prevents overfitting that can occur when directly using this data to train the model when data on the target singer is very limited. The warm-up phase uses the VCTK dataset, which contains 109 different speakers with diverse genders, accents, and language backgrounds, including British English, American English, and Australian English. The total training duration is approximately 44 hours.

[0302] (2) Pre-training stage:

[0303] The purpose of the pre-training phase is to train the model more deeply on singing data, thereby capturing the unique patterns and characteristics of singing speech. This phase helps the model learn singing characteristics such as pitch variation, rhythm, and singing style. The pre-training phase uses approximately 90 hours of singing speech data from a mixed dataset consisting of Opencpop, NUS48E, OpenSinger, and M4singer.

[0304] (3) Adaptation stage:

[0305] The core purpose of the adaptation phase is to adapt the model specifically to the target singer's data, which is particularly important when data on the target singer is limited. This adaptation phase allows the model to apply the knowledge acquired during pre-training to the target singer's specific timbre and style, thereby generating singing voice that matches the target singer's timbre. During this phase, data augmentation based on velocity perturbation is also incorporated, using an augmented dataset twice the size of the original dataset for fine-tuning to mitigate potential overfitting of the singing voice model to limited data. For the in-domain SVC task, the training data was provided by one male and one female singer, totaling approximately 0.4 hours. For the out-of-domain SVC task, the training data was provided by one male and one female singer, totaling approximately 0.2 hours.

[0306] Dataset for singing conversion model training

[0307]

[0308] Model implementation details

[0309] For feature extraction, a medium-sized Whisper model is used to extract 512-dimensional bottleneck features from the 20th layer. The AST model is used with a window size (frame length) of 1024 and a frame shift size of 256 to extract F0 and pitch contours from the waveform, and the energy contour is calculated with the same parameters. During the training process, the pitch contour is randomly transposed in the range of C1 to B6 for data enhancement, thereby obtaining the singing style characteristics. The hidden layer of the LSTM network is set to 128 in voiceprint feature extraction. In the singing voice conversion model, the parameter settings of the prior encoder, posterior encoder and decoder all follow the VITS model. During the training process of the SVC model, the warm-up phase is performed for 400,000 steps, the pre-training phase is performed for 200,000 steps, the batch size is 16, and the training is performed for 100,000 steps in the adaptation phase. The parameter settings of data enhancement follow NANSY. The initial learning rate is set to 1×10 -4 , set the parameters of Adam optimizer to β1=0.8, β2=0.99 to optimize the singing voice conversion model.

[0310] Experimental Environment 2

[0311] Experimental environment Specific content Graphics card NVIDIA GeForce RTX 4090 processor 12th Gen Intel(R) Memory 32GB, 256 SSD operating system Windows 11 Professional 64-bit, Ubuntu Acceleration Tools CUDA 11.8 + CUDNN 8.6 Deep Learning Framework PyTorch 2.0 Development Tools Python 3.9.0

[0312] Comparison of ablation experiment results

[0313] The present invention uses the objective evaluation index MCD to measure the timbre similarity between the generated singing voice and the target singing voice, FPC to measure the similarity of the generated singing voice with the target singer in singing style, and MOSNet as a naturalness mean opinion score (MOS) predictor to measure the naturalness of the generated singing voice. The smaller the MCD value, the higher the timbre similarity between the generated singing voice and the target singing voice. An MCD value less than or close to 6dB usually means a very close timbre. A higher FPC value means that the generated singing voice and the target singing voice are more similar in singing style. When the FPC value is 1, it means that the generated singing voice and the target singing voice are completely consistent in singing style. The naturalness mean opinion score (MOS) ranges from 1 to 5 points.

[0314] In order to verify the effectiveness of the singing voice conversion model of the present invention, four ablation experiments were conducted. First, the singing style feature extraction module was deleted, and only the bottleneck feature, F0, and voiceprint feature were input into the model, which is expressed as w / o Style. Secondly, the pitch shifter module was deleted, which is expressed as w / o Shifter. Thirdly, the improved decoder was replaced with the original decoder based on HIFI-GAN in the VITS model. The sinusoidal excitation signal in the fundamental frequency was no longer extracted, and only the simple F0 was used, which is expressed as D-hifi. Finally, in the three-stage training strategy, the warm-up stage was deleted and replaced with a two-stage training strategy consisting of pre-training and adaptation, which is expressed as w / o Warm-up. The ablation experiment used the SVCC 2023 dataset as the test set.

[0315] Comparison of ablation experiment results

[0316]

[0317] Based on the ablation experimental data and Figure 10From the visual ablation experimental data in , we can see that when there is no input of singing style features, the MCD value does not change significantly, but the FPC value drops by 0.038, and the drop is the largest, and the naturalness also decreases accordingly, indicating that the extraction and input of explicit singing style features will improve the singing voice conversion model's performance in converting singing styles, thereby improving the overall similarity; when there is no pitch shifter, the MCD value still does not change significantly, the FPC value drops slightly, and the naturalness drops by 0.38. The significant drop in naturalness indicates that the pitch shifter can adjust the singing pitch range to the target singer's natural pitch range without changing the overall trend of the singing pitch, thereby improving the singing style. The naturalness of the converted singing voice is affected; when the improved decoder is replaced, the MCD value increases by 0.411, the FPC value decreases by 0.003, the naturalness decreases by 0.17, and the degree of similarity and naturalness decreases, indicating that the original HIFI-GAN decoder is insufficient in modeling the harmonic structure of the singing voice, which is prone to pitch jumps, resulting in unsatisfactory similarity and naturalness of the converted singing voice. The improved neural filter decoder helps to enhance harmonic coherence; similarly, in the absence of a warm-up stage, the naturalness and naturalness will also be greatly reduced. This is because the SVC model lacks sufficient training, which makes the model unable to generate the target singing voice robustly and with high quality.

[0318] Comparison of experimental results of different SVC models

[0319]

[0320] From experimental data and Figure 11 Visualized experimental data in the , show that among the five singing voice conversion methods, DiffSVC achieved the best MCD value (6.126), 0.002 lower than the proposed model (6.128). Analysis reveals that the audio in the mixed dataset of NUS48E and M4singer contains a large number of high-frequency harmonics and exhibits cross-lingual characteristics. DiffSVC has a stronger local fit for these specific acoustic scenarios. While the generalized architecture employed by the proposed model slightly increases spectral reconstruction error, it effectively avoids overfitting, as evidenced by the stability of the naturalness metric (±0.08) across datasets. Regarding FPC, the proposed model achieved the highest result (0.972), demonstrating its significant advantage in learning singing styles and further demonstrating the necessity of explicit singing style modeling. Regarding the naturalness of the converted singing voice, the proposed model continues to outperform the other models, achieving a score of 4.02, further demonstrating that the pitch shifter, improved decoder, and three-stage training strategy can effectively improve the naturalness of the converted singing voice. Comprehensive experimental results show that the singing voice model of the present invention shows better singing voice conversion performance in terms of similarity and naturalness than the other three models.

[0321] The seemingly limited improvement in each model's metrics stems primarily from the characteristics of the metrics themselves: When the MCD value approaches the theoretical lower limit of signal reconstruction (6dB), seemingly minor fluctuations may actually reflect the model's optimization of acoustic details (such as formants and timbre transitions). Even a 0.1 improvement may require precise elimination of minor distortions (such as plosives or vibrato), which is crucial for improving the listening experience. When the FPC value is in the range of 0.9 to 1, every 0.01 gain is required to more accurately capture intonation fluctuations (such as emotional emphasis). In practical scenarios, the 0.029 higher FPC value of the proposed model compared to the baseline model DiffSVC can significantly reduce the harshness of the converted singing voice. The theoretical maximum score for naturalness is 5.0, but in practice, human perception of speech naturalness has a threshold effect. When the score exceeds 4.0, subtle differences may be overlooked by the rater, making the improvement difficult to intuitively reflect. In addition, the standard deviation of the naturalness of the model in this chapter (±0.08) is significantly lower than that of other models, indicating that its generated results are more stable and reliable. Although this implicit improvement does not significantly increase the mean, it enhances the practicality of the model and can effectively reduce occasional distortion.

[0322] The present invention achieves fine acoustic feature decoupling, and on this basis, innovatively designs a pitch-energy cascade singing style conversion encoder to explicitly model the vibrato characteristics and dynamic energy profile of the target singer, effectively capturing the personalized singing style. In terms of model architecture, the harmonic coherence is enhanced by integrating a neural source filter into the decoder, and a discriminator strategy of multi-period and multi-scale joint optimization is adopted to significantly improve the authenticity of pitch periodicity and spectral details. In addition, a variance-driven pitch shifter is introduced to dynamically adjust the pitch distribution to adapt to the natural range of the target singer and avoid the problem of pitch distortion during cross-domain conversion. In terms of model training, a three-stage training strategy is used to further improve the quality of generated singing.

[0323] Experimental results show that the proposed method achieves an MCD value of 6.128, an FPC value of 0.972, and a naturalness of 4.02±0.08 on the NUS48E and M4singer mixed test sets. Compared with the baseline models UCD-SVC, So-VITS-SVC, Fast SVC, and DiffSVC, the MCD value is reduced by up to 10.70%, and the FPC value and naturalness are improved by 8.60% and 14.20%, respectively. Compared with the classic conversion model based on the VITS framework, the timbre similarity, singing style similarity, and naturalness are improved by 2.36%, 5.31%, and 4.14%, respectively. Ablation experiments further verified the effectiveness of key modules. Replacing the improved decoder and missing the warming stage will lead to varying degrees of reduction in similarity and naturalness, indicating the effectiveness of the improved decoder in harmonic modeling and the importance of sufficient training for generating robust singing voices; removing singing style features will cause the FPC value to drop by 0.038, and missing the pitch shifter will reduce the naturalness by 0.38, which fully demonstrates the important role of multi-level feature fusion and adaptive pitch adjustment in the quality of singing voice conversion.

Claims

1. A singing voice conversion method based on VITS, characterized in that: The following steps are included: Step 1: Using the Whisper shallow encoder, the automatic singing transcription model (AST), and the AutoVC-based pitch-energy encoder, we extract bottleneck features, fundamental frequency, singing style features, and linear spectrograms from the source singer's voice. These serve as input for the VITS-based singing voice conversion model. Combined with the target singer's voiceprint features, the singing voice is reconstructed using the VITS-based singing voice conversion model to obtain the converted target singer's singing voice; Step 2: The pitch shifter is used to adjust the pitch according to the pitch difference between the source singer and the target singer. The bottleneck feature, fundamental frequency, and singing style feature are used as inputs of the singing voice conversion model to generate a target singing voice with the characteristics of the target singer.

2. A singing voice conversion method based on VITS according to claim 1, characterized in that, In step one, the Whisper shallow encoder is used to extract bottleneck features, capture voice content features unrelated to the singer, and suppress source timbre interference; the automatic singing transcription model AST is used to extract fundamental frequency and capture the continuous pitch changes of singing voice; the AutoVC-based pitch-energy encoder is used to extract explicit singing style features and explicitly model the singing style.

3. A singing voice conversion method based on VITS according to claim 2, characterized in that, In step 1, the process of extracting bottleneck features from the shallow encoder layer of the Whisper model is as follows: (1) Normalize the spectrogram obtained by inputting the original audio so that its value range is between [-1, 1] and ensure that the mean of the input data is close to 0; (2) The Whisper model uses a Transformer-based encoder to process audio features; After preliminary processing through a series of convolutional layers, the audio features are fed into a Transformer encoder, which then processes them through multiple layers of self-attention and feedforward networks to generate a high-level representation of the audio. The output of each layer contains increasingly abstracted audio information. (3) The bottleneck features of the Whisper model are extracted from the middle and shallow layers of the Transformer encoder.

4. A singing voice conversion method based on VITS according to claim 2, characterized in that, In step 1, voiceprint feature extraction uses Mel-Spectrogram, and the extraction process is as follows: The time domain signal x(t) is directly obtained from the time domain waveform of the input original audio. The time domain signal x(t) is divided into frames through STFT and the spectrum X(m,k) of each frame is calculated, as shown in formula (1): Where m represents the frame index, k represents the frequency index, w(n) represents the window function, using Hamming window, and N represents the number of FFT points. Then square the spectrum amplitude to obtain the power spectrum P(m,k) as shown in formula (2): P(m,k)=|X(m,k)| 2 (2) The calculation method of Mel frequency scale description is shown in formula (3): Where f represents the linear frequency, f mel Indicates the corresponding Mel frequency; On the Mel frequency scale, the Mel filter uses a triangular window function to weight the power spectrum, and the filter response is calculated by the triangular function, as shown in formula (4): where f i+1 , f i+2 Respectively represent the boundary frequencies of the two adjacent filters of the i filter; Triangular filter bank H i (k) The formula for weighting the power spectrum P(m,k) is shown in formula (5): Among them, S(m,i) represents the output of the mth frame on the i-th filter; The dynamic range of the weighted power spectrum is compressed using logarithmic transformation, as shown in formula (6): Log-Mel(m,i)=log(S(m,i)+∈) (6) Among them, ∈ is a smoothing constant to prevent zero values; Finally, the features of each frame are concatenated into a two-dimensional matrix to form a Mel-Spectrogram; the filter output is normalized as shown in formula (7): Among them, F(m,k) represents the value of Mel-Spectrogram in the mth frame and the kth dimension, μ k represents the mean of the feature in the kth dimension, σ k Represents the standard deviation of the feature in the kth dimension; PCA is used to reduce the data dimension and extract the characteristic component matrix of each Mel-Spectrogram group, which consists of n components, each component has d variables, and the sample matrix X is shown in formula (8): Calculate the mean and variance of each column and perform normalization, as shown in Equation (9), Equation (10), and Equation (11): in, represents the mean of the kth column of the feature matrix, represents the variance of the kth column of the feature matrix; Then calculate the correlation coefficient matrix and eigenvalue λ of the standardized features k , and sort from large to small, and calculate the cumulative contribution rate of the first m principal components Construct a dimensionality reduction matrix, as shown in formula (12): The reduced feature matrix is used as the input of BiLSTM. The forward and backward LSTMs process the sequence simultaneously, capturing the dependencies between the features of each frame. The hidden state of each frame is calculated in sequence. The output of each frame is the concatenation of the forward and backward hidden states. The final output is used as the input of the subsequent network. The update formulas of Mel-Spectrogram features in LSTM units after dimensionality reduction are shown in equations (13), (14), (15), (16), and (17): f t =σ(W f ·[h t-1 ,x t ]+b f ) (13) i t =σ(W i ·[h t-1 ,x t ]+b i ) (14) h t =o t ⊙tanh(C t ) (17) Among them, f t 、i t 、o t Represent the forget gate, input gate and output gate in the LSTM unit respectively, C t Represents a memory unit, h t represents the hidden state, σ represents the sigmoid function, W f 、W i 、W c represents the weight matrix, b f 、b i 、b c represents the bias term in the forget gate, input gate, and output gate, h t-1 represents the hidden state of the previous time step, x t Represents the input features of the current time step, ⊙ represents the element-by-element matrix multiplication; The forward and backward calculations of the bidirectional LSTM are shown in Equations (18), (19), and (20): Among them, h t Represents the concatenation of the forward and backward hidden states.

5. A singing voice conversion method based on VITS according to claim 2, characterized in that, The explicit singing style feature extraction is achieved through the singing style conversion encoder: The encoder consists of two parts: pitch contour conversion and energy contour conversion; The pitch conversion part is used to convert the pitch contour of the input singer into the pitch style of the target singer, and the energy conversion part adjusts the energy contour based on the pitch conversion to match the style of the target singer; The specific implementation process of the encoder is as follows: Step 1: Use the automatic singing transcription model AST to extract the accurate initial pitch contour p, and use this parameter to calculate the logarithmic scale energy value of each frame of audio signal to obtain the initial energy contour e. The calculation method is shown in formula (21): Among them, x n Represents the sampling point value in the audio frame, N represents the number of sampling points in the frame, and e(t) represents the energy value at time t; Convert p to MIDI embed p e , for non-integer MIDI values, linear interpolation is used to calculate the embedding, converting e into the embedding e e , and is evenly divided into intervals by logarithm; Calculate the reconstruction loss L recon , as shown in formula (22): Among them, BCE stands for cross entropy loss, RMSE stands for root mean square loss, and weight λ b and λ r Control the relative importance of BCE and RMSE; The learning and reconstruction capabilities of vibrato characteristics are used to calculate the vibrato loss L vibr , as shown in formula (23): Among them, p ft represents the spectral features extracted from the true pitch contour by short-time Fourier transform (STFT), represents the spectral features of the generated predicted pitch contour, p vaf vibrato amplitude characteristics that represent the true pitch contour, represents the vibrato amplitude feature for predicting pitch contour, and the weight λ ft and λ vaf Learn and restore the target singer's vibrato characteristics; Step 2: Extract the input pitch or energy embedding into a compact high-dimensional feature and limit the inflow of unnecessary information through the information bottleneck; the encoders of the two parts of the model are consistent, that is, the pitch is embedded in p e and energy embedded in e e Local temporal features are extracted through 1D convolution. Then, four residual blocks, each containing two layers of 1D convolution, normalization, and ReLU activation functions, are used to extract temporal information through a bidirectional LSTM. The output dimension is 2, and downsampling is performed to form an information bottleneck. Step 3: Both the pitch and energy decoders need to accept the output features of their respective encoders and the target singer embedding obtained from the lookup table, upsample them to the same number of original frames, and then concatenate them. The same 1D convolution, residual block, and bidirectional LSTM are used to process the features. Step 4: Convert the embedded matrix output by the pitch contour model decoder The MIDI embedding value of each frame is mapped back to the actual predicted pitch by weighted averaging, and the pitch contour p and the predicted pitch contour are Apply sinc function to perform high frequency filtering to extract vibrato characteristics p sharp , as shown in formula (24): p sharp =p-sinc(i)*p (24) Where * represents convolution. Then extract the spectrum loss through STFT As shown in formula (25), ensure that the frequency distribution conforms to the target singer's style: The amplitude feature loss is obtained by comparing the root mean square error between the true and predicted amplitude profiles. As shown in formula (26), the vibrato strength and weakness characteristics of the target singer are captured; The smoothness loss is obtained by measuring the smoothness of the amplitude change using the first-order difference As shown in formula (27), the artificial sense caused by sudden amplitude changes is avoided; Finally, the total vibration loss is integrated As shown in Equation (28), the pitch contour generated by the model is guaranteed to be and energy profile Match the target style; The weight parameter λ ft =λ vaf =λ smooth =0.1; Decoder output It is the final result after transformation; This completes the transformation of performance style and obtains the pitch contour that highlights the performance style. and energy profile 6. A singing voice conversion method based on VITS according to claim 1, characterized in that, The singing voice conversion model includes four parts: a priori encoder, a posteriori encoder, a decoder and a discriminator; The posterior encoder and decoder work together through a self-reconstruction mechanism to simulate the hidden representation z of the singing waveform y and predict the original singing waveform. The discriminator performs adversarial training on the generated singing waveform to ensure waveform generation quality. The a priori encoder captures the pitch changes and language content characteristics of the source singing voice and integrates them with the timbre of the target singer. The a priori encoder and posterior encoder are connected using a reversible normalization flow. The specific implementation methods of the four parts of the prior encoder, the posterior encoder, the decoder and the discriminator are as follows: (1) A priori Encoder Using a multi-layer Transformer, the input feature c is transformed using its self-attention mechanism and multi-head attention mechanism. bnf 、c f0 、c style and c vpf Efficient fusion is performed at multiple levels, and then the prior distribution p(z|c bnf ,c f0 ,c style ,c vpf ), and finally outputs the latent variable z, which is used together with the subsequent decoder to generate the final singing waveform; Bottleneck features extracted by Whisper model (using c bnf Representation), F0 feature c extracted by automatic singing transcription model f0 , Singing style features extracted based on AutoVC style And voiceprint features c vpf ; (2) Posterior Encoder The non-causal WaveNet structure is used for construction. The linear Mel spectrogram generated by the singing waveform y is used as input. The residual block in the WaveNet structure is used to learn features at multiple time steps to capture the local dependencies of the input singing. The convolutional layer is used to output the latent variable z, which follows the posterior distribution p(z|y). During the training process, the output of the prior encoder and the posterior encoder are subjected to KL divergence loss calculation to narrow the gap between the prior distribution and the posterior distribution. Minimizing the KL divergence loss helps to make the output of the prior encoder consistent with that of the posterior encoder. (3) Decoder The decoder's task is to generate the target singing waveform based on the hidden representation z The decoder part of the VITS architecture is built based on HIFI-GAN, and the decoding process is implemented using a solution combined with a neural source filter; The improved decoder mainly consists of a multi-receptive field fusion module, a source module, and a filter module; The multi-receptive field fusion module is derived from HIFI-GAN, and the source module converts F0 into an excitation signal e by combining a sine wave. t , which is defined as shown in formula (29). When the frequency is not zero, that is, in the sound stage, the excitation signal e t It is composed of F0 and harmonics. In the silent stage, the excitation signal e t is composed of Gaussian noise; Among them, α represents the amplitude coefficient, n t represents Gaussian random noise, σ represents the standard deviation of Gaussian random noise, N s represents the waveform sampling rate, φ∈[-π,π] represents the random initial phase; In the filter module, the non-autoregressive dilated convolutional network generates the excitation signal e t The high-pass filter processes the silent part, or the noise part, of the signal to preserve the high-frequency components. The low-pass filter processes the vocal part, or the harmonic part, to preserve the low-frequency components. Through a more granular audio modeling method, the pitch is precisely controlled, and a singing voice with coherent pitch changes is naturally generated. (4) Discriminator The discriminator uses a combination of multi-cycle discriminator and multi-scale discriminator. In the singing voice conversion model, the training of the discriminator is adversarial, and the generator part is played by the decoder to generate the target singing waveform close to the real singing voice. The task of the discriminator is to determine whether the generated singing waveform is natural and consistent with the timbre of the target singer, ensuring that the generated singing voice is as close to the target singer as possible.

7. A singing voice conversion method based on VITS according to claim 6, characterized in that, In the multi-period discriminator, the input audio waveform is divided into subsequences according to different periods and input into multiple discriminator branches MPD respectively. i , each branch outputs two types of information: the probability value P of whether the input is real audio i and the intermediate layer features F for feature matching i , then assign weights to the losses of each cycle branch, and sum them up as the total loss L MPD , as shown in formula (30): Among them, L adv,i represents the independent adversarial loss of each cycle branch, w i Indicates the allocation weight; The multi-scale discriminator analyzes different frequency bands to evaluate the low-frequency and high-frequency parts of the audio. This multi-scale discriminant approach helps the generator adjust the details of the generated audio and optimize it in different frequency bands. In the multi-scale discriminator, the input audio is converted into branches of different resolutions by downsampling at different times, and input into multiple discriminator branches MSD respectively. i , each scale branch outputs a probability value P i and the intermediate layer features F i , then assign weights to the losses of each scale branch, and sum them up as the total loss L MSD , as shown in formula (31): Among them, L adv,i represents the independent adversarial loss for each scale branch; Each multi-cycle branch and multi-scale branch intermediate feature map participates in the feature matching loss L FM The calculation of is shown in formula (32): in, represents the l-th layer feature map of the i-th discriminator, C i and H i represents the number of channels and feature map length, ‖*‖1 represents the L1 norm; Final discriminator loss L total It is the weighted sum of MPD loss, MSD loss and feature matching loss, as shown in Equation (33): THE total =L MPD +L MSD +λL FM (33)。 8. A singing voice conversion method based on VITS according to claim 1, characterized in that, The working principle of the pitch shifter in step 2 is divided into the following steps: Step 1: Determine the pitch P of the source singer in a given song or sentence s , the target singer's pitch P t , and calculate the deviation ΔP between the source pitch and the basic pitch, as shown in formula (34): ΔP=P s -P t (34) Step 2: Calculate the source singer's pitch variance and the target singer's pitch variance As shown in formula (35) and (36); Among them, P s,i and P t,i Represent the pitch samples of the source singer and the target singer respectively, μ s and μ t Represents the mean of the source singer and the target singer respectively, N s and N t Represents the number of samples of source singer and target singer respectively; Step 3: Adjust the source singer's pitch based on the difference and variance between the source singer and the target singer to make it closer to the target singer's pitch. The adjusted pitch P p The calculation method of is shown in formula (37): Among them, α controls the adjustment range of the pitch difference. If α is large, the source pitch will shift to the target pitch to a greater extent. β controls the adjustment range of the variance difference. If β is large, the source singer's pitch range will be closer to the target singer's pitch range to a greater extent, avoiding the influence of too small or too large variance. Step 4: Adjusted Pitch P p It is passed to the singing voice conversion model to provide fundamental frequency parameters that match the target singer's vocal range; the target timbre bound to the language content and rhythm provided by the bottleneck feature, the dynamic singing style characteristics injected by the singing style feature, and the voiceprint feature is fused through the multi-layer Transformer prior encoder of the singing voice conversion model, and then the decoder generates the final singing waveform of the target singer for a given song. The adjustment of the pitch shifter ensures that the generated target singing voice is consistent with the target singer's performance in pitch, thereby improving the naturalness and similarity of the conversion result.

Images