Singing sound conversion method and singing sound conversion system

Abstract

The invention relates to the technical field of audio processing, in particular to a song conversion method and system, and the method comprises the steps: carrying out the human voice separation of obtained original audio data, and obtaining clean human voice data; slicing the clean human voice data, and removing silent voice to obtain human voice slicing data; taking the human voice slice data as a training data set, constructing a singing sound conversion model, and training the singing sound conversion model through the training data set by taking minimization of a value of a loss function as a target to obtain a trained singing sound conversion model; and inputting audio data to be converted into the trained singing sound conversion model to obtain a final target singing sound. According to the method, fine audio preprocessing, an innovative model architecture and feature extraction method and flexible loss function design are integrated, and efficient and high-quality singing sound conversion is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a singing voice conversion method and a singing voice conversion system. Background Art

[0002] Singing voice conversion technology is a technology that can convert the timbre of a source singer into the timbre of a target singer. Its core lies in achieving accurate timbre conversion while keeping the singing content unchanged. This technology has shown great application potential and value in many fields such as film and television music production, artistic creation, entertainment industry, and human-computer interaction. Through singing voice conversion, producers can easily replace the voice of a singer with the voice of another singer to meet specific creative needs or achieve personalized customization.

[0003] Among existing singing voice conversion technologies, efficient vocoders such as WaveNet and Parallel WaveGAN are widely used to generate high-quality audio. These vocoders model audio data through deep learning models and are able to generate audio with naturalness and realism. However, although these technologies have made significant progress in sound quality, they still face some challenges and limitations: First, insufficient vocal separation and preprocessing: The original audio data usually contains multiple components such as accompaniment, reverberation, inhalation sound, and environmental noise, which interfere with the effective extraction of vocals. Existing technologies often have deficiencies in vocal separation and preprocessing, resulting in low quality of extracted vocal data, which in turn affects the subsequent singing voice conversion effect.

[0004] Second, computational complexity and training stability: Models such as WaveNet use an autoregressive structure, which means that the process of generating audio needs to be carried out step by step, resulting in low computational efficiency. Although Parallel WaveGAN improves the generation speed, it still requires complex training and reasoning processes. In addition, the model is prone to falling into a local optimal solution during training, resulting in unstable generated audio quality.

[0005] Third, the limitations of emotional expression and timbre conversion: Existing singing voice conversion technologies can only achieve timbre conversion, but cannot fully preserve and transmit the emotional and style characteristics of the source singer. This leads to deficiencies in the emotional expression of the generated audio.

[0006] Fourth, model complexity and data dependence: The existing singing voice conversion model has a complex structure and is highly dependent on data, requiring a large amount of high-quality data for training. This increases the cost and difficulty in practical applications.

[0007] Therefore, in order to overcome the shortcomings of existing technologies and improve the sound quality, stability and emotional expression of singing voice conversion, the industry needs to continuously explore new singing voice conversion methods and algorithms. By optimizing the model structure, improving the training strategy and introducing technical means such as emotional feature extraction, a more efficient, accurate and natural singing voice conversion effect can be achieved. This will inject new vitality and impetus into the development of film and television music production, artistic creation, entertainment industry and human-computer interaction. Summary of the invention

[0008] To solve the above technical problems, the present invention provides a singing voice conversion method, comprising the following steps: S1: performing vocal separation on the acquired original audio data to obtain clean vocal data; S2: Slice the clean vocal data to remove silent sounds, and obtain vocal slice data; S3: Using the vocal slice data as a training data set, constructing a singing voice conversion model, and training the singing voice conversion model with the training data set as the goal to minimize the value of the loss function to obtain a trained singing voice conversion model; wherein the loss function includes Mel feature loss, pitch loss and KL divergence loss; S4: inputting the audio data to be converted into the trained singing voice conversion model to obtain the final target singing voice; wherein, in step S4, the steps of obtaining the final target singing voice are as follows: S41: extracting features of the audio data to be converted respectively to obtain embedded semantic features, pitch features and mel-spectrogram features; S42: Processing the embedded semantic features and the pitch features to extract the audio text of the speaker; processing the mel-spectrogram features to obtain the timbre features of the speaker; S43: Perform audio synthesis on the timbre feature and the audio text to obtain a final target singing voice.

[0009] In one embodiment of the present invention, in S1, the steps of obtaining clean vocal data are as follows: S11: Separate the accompaniment of the original audio data to obtain initial vocal data; S12: Based on the initial human voice data, remove reverberation, inhalation sound and environmental noise to obtain the clean human voice data.

[0010] In one embodiment of the present invention, in S42, the step of extracting the audio text of the speaker is as follows: S421: constructing an audio content extraction model, and iteratively training the audio content extraction model to obtain a trained audio content extraction model, including: The audio content extraction model includes a BERT model and an Adapter module, freezing the parameters of the multi-head attention layer and the feedforward neural network layer in the BERT model, and only iteratively updating the parameters of the Adapter module; S422: Use the trained audio content extraction model to process the embedded semantic features and the pitch features, perform feature extraction through the BERT model and the Adapter module, and obtain the audio text of the speaker.

[0011] In one embodiment of the present invention, in S422, the audio text of the speaker is obtained. : , in, Represents the input of the Adapter module; Indicates input The two-dimensional weight matrix for dimensionality reduction; Indicates input The bias matrix for dimensionality reduction; Indicates the weight matrix that converts the reduced-dimensional features into the original dimension; The bias matrix is ​​used to convert the reduced features into the original dimensions.

[0012] In one embodiment of the present invention, the Adapter module is composed of a two-layer linear network, a ReLU activation function and a residual network.

[0013] In one embodiment of the present invention, in S42, the step of obtaining the timbre characteristics of the speaker is as follows: S423: constructing a timbre feature extraction model, wherein the timbre feature extraction model includes a stacked causal convolution layer, a residual connection layer, and a gated activation module; S424: at each time node, obtaining the temporal dependency feature in the mel-spectrogram feature through the causal convolution layer and the gated activation module, calculating the probability distribution value of the next sampling point based on the current mel-spectrogram feature and the temporal dependency feature of the previous time node, and sampling to obtain the next sampling point; S425: looping through step S424 to continuously generate new sampling points until a speech feature sequence is obtained; S426: Extracting the timbre features of the speaker based on the speech feature sequence.

[0014] In one embodiment of the present invention, the loss function is expressed as: , Among them, L is the total loss function value, is the attenuation coefficient, represents the pitch loss, represents the KL divergence loss, represents the Mel feature loss.

[0015] In one embodiment of the present invention, the Mel feature loss The calculation method is: , in, represents the actual mel-spectrogram features of the input; Represents the Mel spectrogram features of the audio generated by the model; The pitch loss The calculation method is: , in, Represents the embedded semantic features after frequency conversion; express go through f0 The predicted embedded semantic features obtained by the decoder; n represents the number of samples; The KL divergence loss The calculation method is: , in, and is the embedded representation of the i-th sample after enc_p encoding, where Represents the output of the first half of the encoding result, Represents the output of the second half of the encoding result; and is the embedded representation and mask representation of the i-th sample after enc_q encoding, , Represents the embedding representation of the i-th audio sample generated by the HuBERT model; represents the embedding representation obtained by the flow model, .

[0016] Based on the same inventive concept, the present invention also provides a singing voice conversion system, which includes the following modules: A clean vocal data acquisition module is used to separate the vocals from the acquired original audio data to obtain clean vocal data; A vocal slice data acquisition module is used to slice the clean vocal data, remove silent sounds, and obtain vocal slice data; A singing voice conversion model construction module, used to use the vocal slice data as a training data set to construct a singing voice conversion model, with the goal of minimizing the value of a loss function, and to train the singing voice conversion model through the training data set to obtain a trained singing voice conversion model; wherein the loss function includes Mel feature loss, pitch loss and KL divergence loss; The audio synthesis module is used to input the audio data to be converted into the trained singing voice conversion model to obtain the final target singing voice; wherein the method for obtaining the final target singing voice is as follows: Feature extraction is performed on the audio data to be converted respectively to obtain embedded semantic features, pitch features and Mel-spectrogram features; the embedded semantic features and the pitch features are processed to extract the audio text of the speaker; the Mel-spectrogram features are processed to obtain the timbre features of the speaker; the timbre features and the audio text are subjected to audio synthesis to obtain the final target singing voice.

[0017] The above technical solution of the present invention has the following advantages compared with the prior art: 1. Efficient vocal separation and preprocessing: The present invention finds the optimal combination through the combined operation of multiple UVR5 models. Thus, high-quality clean vocal data is separated from the original audio data, and then conventional slicing processing is performed to remove silent sounds, etc., which further effectively reduces data redundancy, improves model training efficiency and the real-time performance of singing voice conversion, and provides a high-quality data foundation for subsequent processing.

[0018] 2. Optimize the loss function of model training: measure the difference between the audio generated by the model and the real audio from multiple angles, including Mel loss, pitch loss, and KL divergence loss, to ensure that the audio generated by the model keeps getting closer to the real audio as the training progresses. At the same time, considering the different degrees of dependence on each loss at different stages of training, an adjustment attenuation coefficient is introduced to linearly attenuate the weight of the pitch loss function in the later stages of training. By balancing the impact of different loss terms during the training process, the training process of the final model can converge smoothly.

[0019] 3. Efficient fine-tuning: Unlike traditional fine-tuning of all model parameters, the present invention freezes the parameters of the multi-head attention and feedforward layers in BERT and adds an Adapter module for training. The experimental verification results reveal the outstanding performance of the present invention in enhancing audio processing efficiency, which is specifically reflected in the significant improvement in model training speed. When the model loss reaches the preset threshold, the average time consumed by the present invention is reduced by 15% compared to the traditional fine-tuning method, which fully demonstrates its efficiency and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 It is a flow chart of a singing voice conversion method provided in the first embodiment of the present invention; Figure 2 1 is a training loss function curve of the singing voice conversion model in the first embodiment of the present invention, wherein (a) represents KL divergence loss, (b) represents mean square error (mse) loss, and (c) represents L1 loss; Figure 3 The performance of the singing voice conversion model proposed in the present invention is compared with that of five other existing singing voice conversion models (RVC-Flow, SoVITS-Flow, SoVITS-Diff, CoMoSVC, and Amphion) based on five evaluation index values. Figure 4 It is a structural schematic diagram of a singing voice conversion system provided in the second embodiment of the present invention; Explanation of the accompanying drawings in the specification: 100, clean vocal data acquisition module; 200, vocal slice data acquisition module; 300, singing voice conversion model construction module; 400, audio synthesis module. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0022] Embodiment 1

[0023] Reference Figure 1 As shown, the present invention provides a singing voice conversion method, the method comprising the following steps: S1: Use the UVR5 model to separate the human voice from the original audio data to obtain clean human voice data; S2: Slicing the clean vocal data using a slicer tool to remove redundant silent sounds to obtain vocal slice data; S3: Using the vocal slice data as a training data set, constructing a singing voice conversion model, with the goal of minimizing the value of the loss function, training the singing voice conversion model through the training data set to obtain a trained singing voice conversion model; S4: Input the audio data to be converted into the trained singing voice conversion model to obtain the final target singing voice.

[0024] Furthermore, in step S1, three UVR5 models are used to separate the acquired original audio data into human voices to obtain clean human voice data. The specific steps of the method are as follows: S11: Separate the accompaniment of the original audio data through the MDX-Net Kim Vocal 2 model to obtain initial vocal data; S12: Based on the initial vocal data, the VR Architecture 9_HP2-UVR model is used to remove reverberation, and the VR Architecture UVR-De-Echo-Aggeressive model is used to remove the inhalation sound and environmental noise in the singing voice to obtain the clean vocal data.

[0025] Further, in step S4, the audio data to be converted is input into the trained singing voice conversion model to obtain the final target singing voice, and the method steps are as follows: S41: Use the ContentVec model, rmvpe model, and Mel-spectrogram feature extraction algorithm to extract features of the audio data to be converted, and obtain embedding features, pitch ( f0 ) features and mel-spectrum features; S42: The main structure of feature extraction of Enc_p encoder is the pre-trained BERT model. First, the main parameters of the BERT model are frozen, and the Adapter module is added to help the model adapt to timbre learning. The embedded semantic features and the pitch features are processed by Enc_p encoder to extract the audio text of the speaker; the Mel spectrogram features are processed by Enc_q encoder to obtain the timbre features of the speaker; S43: The timbre features are processed through the Flow model, and the processed timbre features and the audio text are synthesized through the Nsf_vocoder vocoder to obtain the final target singing voice.

[0026] Specifically, in step S42, the steps of extracting the audio text of the speaker are as follows: S421: constructing an audio content extraction model, and iteratively training the audio content extraction model to obtain a trained audio content extraction model, including: The audio content extraction model includes a BERT model and an Adapter module, freezing the parameters of the multi-head attention layer and the feed-forward neural network layer in the BERT model, and only iteratively updating the parameters of the Adapter module; wherein the Adapter module is composed of a two-layer linear network, a ReLU activation function, and a residual network; S422: Use the trained audio content extraction model to process the embedded semantic features and the pitch features, perform feature extraction through the BERT model and the Adapter module, and obtain the audio text of the speaker. : , in, Represents the input of the Adapter module; Indicates input The two-dimensional weight matrix for dimensionality reduction; Indicates input The bias matrix for dimensionality reduction; Indicates the weight matrix that converts the reduced-dimensional features into the original dimension; The bias matrix is ​​used to convert the reduced features into the original dimensions.

[0027] In this embodiment, in S42, the embedded semantic features and the pitch features are processed by the Enc_p encoder to obtain the timbre features of the speaker, and the specific steps are as follows: S423: constructing a timbre feature extraction model based on the WaveNet network structure, wherein the timbre feature extraction model includes a stacked atrous causal convolution layer, a residual connection layer, and a gated activation module; S424: At each time node, the mel-spectrogram feature is used as a conditional input, and the temporal dependency feature in the mel-spectrogram feature is obtained through the causal convolution layer and the gated activation module. Based on the current mel-spectrogram feature and the temporal dependency feature of the previous time node, the probability distribution value of the next sampling point is calculated, and the next sampling point is sampled; S425: looping through step S424 to continuously generate new sampling points until a complete speech feature sequence is obtained; S426: Extracting the timbre features of the speaker based on the speech feature sequence.

[0028] In summary, by extracting embedded semantic features, pitch features, and mel-spectrogram features and processing them separately, the audio text and timbre features of the speaker are extracted. This processing method not only achieves the conversion of timbre, but also retains the emotional and style characteristics of the source singer as much as possible, thereby improving the naturalness and realism of the generated audio.

[0029] In this embodiment, the Flow model shows wide adaptability to data distributions of various complexity, and can achieve fine control over the generation process in data generation tasks. The core of the model adopts the Residual Coupling Block module design based on the concept of Residual Network (ResNet). Its working mechanism is to divide the input data into two subsets, apply a transformation operation to one of the subsets, and then merge the transformed subset with the other untransformed subset to construct an efficient reversible mapping mechanism. This design gives the Flow model a powerful ability to process high-dimensional and complex data structures (such as audio signals).

[0030] When training the Flow model, we mainly focus on the Mel feature loss of the audio feature extraction in the enc_q module, the pitch loss of the audio content extraction in the enc_p module, and the KL divergence loss of the generator. Therefore, the expression of the loss function is: , Among them, L is the total loss function value, represents the pitch loss, represents the KL divergence loss, represents the Mel feature loss. is the attenuation coefficient, which shows a linear decreasing trend from 1 to 0 as the number of training rounds increases. However, as the number of training rounds increases, The value of will gradually decrease, resulting in a decrease in the weight of the pitch extraction module in the total loss function. Correspondingly, the loss of the Flow model and the Nsf_vocoder vocoder in generating real audio will account for a larger proportion, thereby guiding the model to pay more attention to the performance improvement of these two modules.

[0031] Among them, the Mel feature loss is calculated based on the L1_Loss loss function : , in, represents the actual mel-spectrogram features of the input; Represents the Mel-spectrogram features of the audio generated by the model; the Mel-spectrogram features help the model reduce the error of the extracted audio features during the training process, thereby improving the prediction performance of the model.

[0032] Constructing pitch loss using mean squared error (mse) , which is calculated as: , in, Represents the embedded semantic features after frequency conversion; express go through f0 The predicted embedded semantic features obtained by the decoder; The KL divergence is used to measure the difference in the results generated by the enc_p module, enc_q module and flow model, so the KL divergence loss The calculation method is: , in, and is the embedded representation of the i-th sample after enc_p encoding, where Represents the output of the first half of the encoding result, Represents the output of the second half of the encoding result; and is the embedded representation and mask representation of the i-th sample after enc_q encoding, , Represents the embedding representation of the i-th audio sample generated by the HuBERT model; represents the embedding representation obtained by the flow model, .

[0033] Audio data of four singers and audio data from the public high-quality Mandarin singing corpus Opencpop are collected as audio datasets for training. The specific content of each audio dataset is shown in Table 1 below.

[0034] Table 1

[0035] According to the above technical scheme, firstly, the human voice is separated from the different types of audio data sets obtained above by the UVR5 model to obtain clean human voice data; secondly, the clean human voice data is sliced ​​by the slicer tool to remove redundant silent sounds to obtain human voice slice data; then, the human voice slice data is used as a training data set to construct a singing voice conversion model, and the singing voice conversion model is trained by the training data set with the goal of minimizing the value of the loss function to obtain a trained singing voice conversion model; finally, the audio data to be converted is input into the trained singing voice conversion model to obtain the final target singing voice.

[0036] When the singing voice conversion method described in the present invention is used to perform model training for different types of audio data sets, the average training time is reduced by 15%, which means that the present invention has significant advantages in improving audio processing efficiency. Specifically, with respect to the audio data set of singer 1, the singing voice conversion method described in the present invention reduces the model training time by 16.3%; with respect to the audio data set of singer 2, the singing voice conversion method described in the present invention reduces the model training time by 14.8%; with respect to the audio data set of singer 3, the singing voice conversion method described in the present invention reduces the model training time by 14.2%; with respect to the audio data set of singer 4, the singing voice conversion method described in the present invention reduces the model training time by 15.4%; with respect to the audio data set of Opencpop, the singing voice conversion method described in the present invention reduces the model training time by 16.7%.

[0037] like Figure 2 As shown, three loss curves are used: KL divergence loss, mse loss (i.e., the " ”), L1 loss (i.e. the “ ”), to comprehensively evaluate the performance of the singing voice conversion model proposed in the present invention. By observing Figure 2 From the experimental results of (a) to (c), we can draw the following conclusions: After the training process advances to a certain stage, the loss value gradually stabilizes, which indicates that the model has successfully reached a convergence state after balancing the influence of different loss terms, that is, a (local) optimal solution has been found. The realization of this state is due to the fine regulation of the training process by the present invention through the optimized loss function.

[0038] Among them, mse loss and L1 loss are used to quantify the gap between the reconstructed audio pitch features and Mel features and the real features, so as to guide the model decoder to generate higher quality sound. As the number of training rounds increases, both loss values ​​show a trend of gradually decreasing and stabilizing. On the one hand, this shows that the generation performance of the model is constantly improving, and on the other hand, it also provides a basis for stopping training at the appropriate training inflection point, so as to achieve the expected effect by optimizing the training time.

[0039] KL divergence loss, as a metric for measuring the difference between two probability distributions, is used to compare the difference between the potential distribution generated by the encoder enc_q, enc_p, and flow modules and the standard normal distribution (mean 0, variance 1). By introducing KL divergence loss, the distribution structure of the latent space can be effectively constrained, thereby improving the generalization ability of the model. In the actual training process, the KL divergence loss shows a slight downward fluctuation trend, which fully proves that the generalization ability of the model in generating audio is continuously improving.

[0040] However, when selecting the appropriate number of iterations to generate the final model, the actual synthesis quality of the audio needs to be considered comprehensively. Therefore, the present invention combines the MSE loss and the L1 loss for comprehensive evaluation to ensure that the obtained model not only has excellent performance, but also can meet the needs of actual application scenarios.

[0041] In order to comprehensively evaluate the effect of singing voice conversion, STOI (Short-Time ObjectiveIntelligibility), PESQ (Perceptual Evaluation of SpeechQuality), FPC (F0 Pearson Correlation, used to evaluate pitch accuracy), CER (Character Error Rate, reflecting audio quality and clarity) and SIM (Similarity Index) were used as evaluation indicators, and the converted singing voice was quantitatively analyzed from different dimensions.

[0042] exist Figure 3 In this paper, five existing singing voice conversion models, namely RVC-Flow, SoVITS-Flow, SoVITS-Diff, CoMoSVC and Amphion, are selected and compared with the singing voice conversion model proposed in the present invention. Among them, the STOI index is used to measure the clarity of the audio. The closer its value is to 1, the closer the clarity is to the original real audio; the PESQ index is used to evaluate the voice quality. The higher its value is, the better the quality of the generated audio is; the FPC index evaluates whether the audio is out of tune by calculating the F0 Pearson correlation coefficient. The larger its value is, the more similar it is to the real audio pitch; the CER index obtains the text information of the audio through the whisper algorithm and calculates the character error rate. The smaller its value is, the higher the quality and intelligibility of the generated audio is; the SIM index is expressed by the embedding of the generated audio through expnet2, and the cosine distance is calculated to evaluate the similarity between the generated audio and the real audio. The larger its value is, the higher the similarity is.

[0043] from Figure 3 The evaluation results show that the singing voice conversion model proposed in the present invention has shown excellent performance in all evaluation indicators, and most indicators are better than other comparison models. This result fully proves the advancement and practicality of the present invention in the field of singing voice conversion, and provides new ideas and methods for the development of singing voice conversion technology.

[0044] Embodiment 2

[0045] Based on the same inventive concept as the method described in Example 1, the present invention also provides a singing voice conversion system. Figure 4 As shown, the system includes the following modules: The clean vocal data acquisition module 100 is used to separate the vocals from the acquired original audio data to obtain clean vocal data; The vocal slice data acquisition module 200 is used to slice the clean vocal data to remove the silent sound and obtain the vocal slice data; The singing voice conversion model construction module 300 is used to use the vocal slice data as a training data set to construct a singing voice conversion model, and to train the singing voice conversion model through the training data set with the goal of minimizing the value of the loss function to obtain a trained singing voice conversion model; wherein the loss function includes Mel feature loss, pitch loss and KL divergence loss; The audio synthesis module 400 is used to input the audio data to be converted into the trained singing voice conversion model to obtain the final target singing voice; wherein the method for obtaining the final target singing voice is as follows: Feature extraction is performed on the audio data to be converted respectively to obtain embedded semantic features, pitch features and Mel-spectrogram features; the embedded semantic features and the pitch features are processed to extract the audio text of the speaker; the Mel-spectrogram features are processed to obtain the timbre features of the speaker; the timbre features and the audio text are subjected to audio synthesis to obtain the final target singing voice.

[0046] A singing voice conversion system proposed in this embodiment is used to implement the aforementioned singing voice conversion method. Therefore, the specific implementation method of the singing voice conversion system can be seen in the embodiment part of the aforementioned singing voice conversion method. For example, the clean vocal data acquisition module 100, the vocal slice data acquisition module 200, the singing voice conversion model construction module 300 and the audio synthesis module 400 are respectively used to implement the corresponding steps S1, S2, S3 and S4 in the singing voice conversion method in embodiment one. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0047] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0048] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0049] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0051] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A singing voice conversion method, characterized in that: The following steps are involved: S1: performing vocal separation on the acquired original audio data to obtain clean vocal data; S2: Slice the clean vocal data to remove silent sounds, and obtain vocal slice data; S3: Using the vocal slice data as a training data set, constructing a singing voice conversion model, and training the singing voice conversion model with the training data set as the goal to minimize the value of the loss function to obtain a trained singing voice conversion model; wherein the loss function includes Mel feature loss, pitch loss and KL divergence loss; S4: inputting the audio data to be converted into the trained singing voice conversion model to obtain the final target singing voice; Among them, in step S4, the steps of obtaining the final target singing voice are as follows: S41: extracting features of the audio data to be converted respectively to obtain embedded semantic features, pitch features and mel-spectrogram features; S42: Processing the embedded semantic features and the pitch features to extract the audio text of the speaker; processing the mel-spectrogram features to obtain the timbre features of the speaker; S43: Perform audio synthesis on the timbre feature and the audio text to obtain a final target singing voice.

2. The singing voice conversion method according to claim 1, characterized in that: In S1, the steps to obtain clean vocal data are as follows: S11: Separate the accompaniment of the original audio data to obtain initial vocal data; S12: Based on the initial human voice data, remove reverberation, inhalation sound and environmental noise to obtain the clean human voice data.

3. The singing voice conversion method according to claim 1, characterized in that: In S42, the steps of extracting the audio text of the speaker are as follows: S421: constructing an audio content extraction model, and iteratively training the audio content extraction model to obtain a trained audio content extraction model, including: The audio content extraction model includes a BERT model and an Adapter module, freezing the parameters of the multi-head attention layer and the feedforward neural network layer in the BERT model, and only iteratively updating the parameters of the Adapter module; S422: Use the trained audio content extraction model to process the embedded semantic features and the pitch features, perform feature extraction through the BERT model and the Adapter module, and obtain the audio text of the speaker.

4. The singing voice conversion method according to claim 3, characterized in that: In S422, the audio text of the speaker is obtained : ， in, Represents the input of the Adapter module; Indicates input The two-dimensional weight matrix for dimensionality reduction; Indicates input The bias matrix for dimensionality reduction; Indicates the weight matrix that converts the reduced-dimensional features into the original dimension; The bias matrix is ​​used to convert the reduced features into the original dimensions.

5. The singing voice conversion method according to claim 3, characterized in that: The Adapter module consists of a two-layer linear network, a ReLU activation function and a residual network.

6. The singing voice conversion method according to claim 1, characterized in that: In S42, the steps of obtaining the timbre characteristics of the speaker are as follows: S423: constructing a timbre feature extraction model, wherein the timbre feature extraction model includes a stacked causal convolution layer, a residual connection layer, and a gated activation module; S424: at each time node, obtaining the temporal dependency feature in the mel-spectrogram feature through the causal convolution layer and the gated activation module, calculating the probability distribution value of the next sampling point based on the current mel-spectrogram feature and the temporal dependency feature of the previous time node, and sampling to obtain the next sampling point; S425: looping through step S424 to continuously generate new sampling points until a speech feature sequence is obtained; S426: Extracting the timbre features of the speaker based on the speech feature sequence.

7. The singing voice conversion method according to claim 1, characterized in that: The expression of the loss function is: ， Among them, L is the total loss function value, is the attenuation coefficient, represents the pitch loss, represents the KL divergence loss, represents the Mel feature loss.

8. The singing voice conversion method according to claim 7, characterized in that: The Mel feature loss The calculation method is: ， in, Represents the actual Mel-spectrogram features of the input; Represents the Mel spectrogram features of the audio generated by the model; The pitch loss The calculation method is: ， in, Represents the embedded semantic features after frequency conversion; express go through f0 The predicted embedded semantic features obtained by the decoder; n represents the number of samples; The KL divergence loss The calculation method is: ， in, and is the embedded representation of the i-th sample after enc_p encoding, where Represents the output of the first half of the encoding result, Represents the output of the second half of the encoding result; and is the embedded representation and mask representation of the i-th sample after enc_q encoding, , Represents the embedding representation of the i-th audio sample generated by the HuBERT model; represents the embedding representation obtained by the flow model, .

9. A singing voice conversion system, characterized in that: The system includes the following modules: A clean vocal data acquisition module is used to separate the vocals from the acquired original audio data to obtain clean vocal data; A vocal slice data acquisition module is used to slice the clean vocal data, remove silent sounds, and obtain vocal slice data; A singing voice conversion model construction module, used to use the vocal slice data as a training data set to construct a singing voice conversion model, with the goal of minimizing the value of a loss function, and to train the singing voice conversion model through the training data set to obtain a trained singing voice conversion model; wherein the loss function includes Mel feature loss, pitch loss and KL divergence loss; The audio synthesis module is used to input the audio data to be converted into the trained singing voice conversion model to obtain the final target singing voice; wherein the method for obtaining the final target singing voice is as follows: Feature extraction is performed on the audio data to be converted respectively to obtain embedded semantic features, pitch features and Mel-spectrogram features; the embedded semantic features and the pitch features are processed to extract the audio text of the speaker; the Mel-spectrogram features are processed to obtain the timbre features of the speaker; the timbre features and the audio text are subjected to audio synthesis to obtain the final target singing voice.

Images