Song conversion method and device based on multi-feature fusion, equipment and storage medium

Abstract

This invention relates to the field of vocal conversion, specifically to a vocal conversion method based on multi-feature fusion. The method extracts multi-dimensional features from the source audio, including HuBERT, fundamental frequency, and volume, and extracts singer embeddings using an embedding extractor. These features are then fused. The fused features are fed into a vocal conversion model for training. The trained model can then transfer the timbre of a target singer to any audio file from multiple source singers' audio datasets, generating audio containing both the given target singer's timbre and the source singers' audio content. This method achieves high-quality, high-stability generation of vocal acoustic features, effectively improving the naturalness and expressiveness of vocal conversion.

Description

Technical Field

[0001] This invention relates to the field of vocal conversion, specifically to a vocal conversion method, apparatus, device, and storage medium based on multi-feature fusion. Background Technology

[0002] Vocal conversion aims to alter the timbre, singing style, and other attributes of a source vocal track to make it sound like it was sung by a target singer, while preserving the original melody and lyrics. This technology has broad application prospects in music production, entertainment applications, and virtual singers. Its core challenge lies in effectively decoupling and remodeling the content information of the source audio with the timbre information of the target voice.

[0003] Traditional singing voice conversion methods typically rely on signal processing or statistical models, such as Gaussian mixture models, but these methods often have limitations in terms of naturalness and sound quality. With the development of deep learning, neural network-based acoustic models, especially sequence-to-sequence models and generative adversarial networks, have become mainstream. These models can learn complex acoustic feature mapping relationships from large amounts of data. However, existing deep singing voice conversion models still face several challenges: First, singing voices are highly dynamic and expressive, with complex variations in pitch, volume, and timbre over time; jointly and meticulously modeling these three aspects is crucial. Second, to generate highly natural and high-fidelity sounds, acoustic models need to predict high-quality Mel spectra, requiring strong sequence modeling capabilities to capture long-range temporal dependencies. Third, in multi-speaker scenarios, models need to effectively and flexibly model and manipulate speaker identity information.

[0004] The Transformer architecture has achieved great success in natural language processing and speech synthesis due to its powerful sequence modeling capabilities and parallel computing efficiency. Applying it to acoustic modeling of singing conversion can better handle long sequences of acoustic features. Although some existing solutions have introduced Transformer, their feature fusion level is usually relatively simple, failing to fully and explicitly model the acoustic factors that are crucial to singing expressiveness, and also failing to specifically design for practical scenarios such as multi-speaker mixing. Summary of the Invention

[0005] Therefore, it is necessary to address the problems in existing singing voice conversion technologies, such as insufficient joint modeling of multi-dimensional acoustic features like pitch and volume, and the need to improve the quality and stability of Mel spectrum generation. This paper proposes a singing voice conversion method based on multi-feature fusion, which can achieve high-quality and high-stability generation of singing voice acoustic features, effectively improving the naturalness and expressiveness of singing voice conversion.

[0006] In one embodiment, the present invention provides a multi-feature fusion singing voice conversion method, comprising: S10, Obtain the audio dataset; wherein, the audio dataset includes the audio data of the source singer and the target singer, and the corresponding singer ID; S20, feature extraction is performed on the audio data of the source singer to obtain the content features, fundamental frequency, and volume of the source singer's audio data; embedding extraction is performed on the singer ID of the target singer to obtain the embedding of the target singer; S30, feature fusion is performed on the content features, fundamental frequency, volume of the source singer and the embedding of the target singer to obtain fused audio feature information; S40, the fused audio feature information is input into the trained singing conversion model, and the Mel spectrum after singing conversion is output.

[0007] Furthermore, S30 includes: S301, normalize the fundamental frequency of the source singer to obtain the normalized fundamental frequency; S302, Project the content features, volume and normalized fundamental frequency of the source singer, and map the content features, volume and normalized fundamental frequency of the source singer to the same dimension. S303, the source singer's content features, volume, fundamental frequency and target singer's embedding obtained after projection processing are added and fused frame by frame to obtain fused audio feature information.

[0008] Furthermore, S40 includes: S401, Perform multi-head self-attention processing on the fused audio feature information, and output the self-attention processed audio feature information; S402, perform feedforward processing on the audio feature information after self-attention processing, and output the encoded audio feature information; S403, input the encoded audio feature information into the multi-head self-attention submodule of the next encoder, and repeat steps S401 to S403 until all encoder processing is completed to obtain the final encoded audio feature information; S404, the final encoded audio feature information is mapped and output to obtain the Mel spectrum after the singing voice is converted.

[0009] Furthermore, S401 includes: S4011, Perform layer normalization processing on the fused audio feature information to obtain the layer normalized fused audio feature information; S4012, Parallel calculation of the correlation weights between all positions in the fused audio feature information after the layer normalization process, generating a context vector of the fused global context; S4013, the context vector of the fused global context is added to the fused audio feature information by residual addition to obtain the first intermediate fused audio feature information; S4014, the obtained first intermediate fused audio feature information is subjected to layer normalization processing, and the self-attention processed audio feature information is output.

[0010] Furthermore, S402 includes: S4021, Nonlinear mapping is performed on the audio feature information after self-attention processing to obtain the mapped audio feature information; S4022, Perform projection dimension transformation on the mapped audio feature information to obtain standard dimension audio feature information; S4023, The residuals of the standard dimension audio feature information and the self-attention processed audio feature information are added to obtain the second intermediate fused audio feature information; S4024, the obtained second intermediate fused audio feature information is subjected to layer normalization processing, and the encoded audio feature information is output.

[0011] Furthermore, in S40, the training process of the singing voice conversion model includes: S01, Obtain the singer's audio dataset and preprocess each audio data in the audio dataset to generate a configuration file; S02, perform feature extraction on each audio data in the configuration file to obtain the singer's content features, fundamental frequency, volume, and true Mel spectrum; and perform embedding extraction on the singer's singer ID to obtain the singer's embedding. S03, interpolate the content features of the singer, and use the same starting frame and frame number to slice the base frequency, volume and the interpolated content features to obtain the sliced ​​content features, base frequency and volume; S04, perform feature fusion on the content features, fundamental frequency, volume and corresponding singer embedding of the sliced ​​singer to obtain the fused audio feature information; S05, input the fused audio feature information into the Transformer module, and output the final encoded audio feature information; S06, the final encoded audio feature information is input to the spectrum prediction output module for mapping output to obtain the predicted Mel spectrum, and the trained singing conversion model is determined based on the loss between the predicted Mel spectrum and the real Mel spectrum.

[0012] In one embodiment, the present invention also provides a multi-feature fusion singing voice conversion device, comprising: a data acquisition module, a feature extraction module, a feature fusion module, and a singing voice conversion output module. The data acquisition module is used to acquire an audio dataset; wherein, the audio dataset includes audio data of the source singer and the target singer, and the corresponding singer ID; The feature extraction module is used to extract features from the audio data of the source singer to obtain the content features, fundamental frequency, and volume of the source singer's audio data; and to extract the singer ID of the target singer to obtain the embedding of the target singer. The feature fusion module is used to perform feature fusion on the content features, fundamental frequency, volume of the source singer and the embedding of the target singer to obtain fused audio feature information. The singing voice conversion output module is used to input the fused audio feature information into the trained singing voice conversion model and output the Mel spectrum after singing voice conversion.

[0013] Furthermore, the feature extraction module is also used to extract the embedding of the source singer ID to obtain the embedding of the source singer; the feature fusion module is also used to fuse the content features, fundamental frequency, volume and embedding of the source singer to obtain fused audio feature information.

[0014] In one embodiment, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the singing voice conversion method described in any of the above embodiments.

[0015] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the singing voice conversion method described in any one of the above embodiments.

[0016] The singing voice conversion method based on multi-feature fusion provided by this invention extracts multi-dimensional features such as HuBERT, fundamental frequency, and volume from the source audio, and extracts the singer ID embedding through an embedding extractor. These features are then fused. The fused features are fed into a singing voice conversion model for training. The trained model can then transfer the timbre of a target singer to any audio file from multiple source singers' audio datasets, generating audio containing both the given target singer's timbre and the source singers' audio content. This invention utilizes a multi-feature fusion mechanism to fuse multi-dimensional acoustic information such as discrete content and continuous prosody, ensuring that all key acoustic factors are utilized during the model's generation process. It effectively models long-distance contextual dependencies in singing voice using Transformer, and through explicit deep feature fusion and precise sequence modeling, significantly improves the accuracy of acoustic feature generation in singing voice conversion, as well as the naturalness, expressiveness, and quality and efficiency of the generated speech. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Furthermore, the accompanying drawings are not drawn to a 1:1 scale, and the relative dimensions of the various components are shown in the drawings only as examples and not necessarily to actual scale.

[0020] Figure 1 This is a flowchart illustrating the singing voice conversion method based on multi-feature fusion provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the feature fusion steps of the singing voice conversion method in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the singing conversion steps of the singing conversion method in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the multi-head self-attention processing steps of the singing voice conversion method in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the feedforward processing steps of the singing voice conversion method in an embodiment of the present invention; Figure 6A schematic diagram of the structure of the singing voice conversion device based on multi-feature fusion provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] In this embodiment of the invention, a singing voice conversion model is first constructed, and then the singing voice conversion model is trained using a dataset containing multi-style singing audio of multiple singers to obtain a trained singing voice conversion model. When performing singing voice conversion, the content features, fundamental frequency, volume of the source singer to be processed and the embedding of the target singer are input into the trained singing voice conversion model to obtain the converted target singing voice.

[0023] Specifically, please refer to Figure 1 , Figure 1 The diagram illustrates a flowchart of a singing voice conversion method based on multi-feature fusion according to an embodiment of the present invention. This method is implemented using a singing voice conversion device, which includes a data acquisition module, a feature extraction module, a feature fusion module, and a singing voice conversion output module. The device performs the following steps: The data acquisition module is used to execute step S10 to acquire an audio dataset; wherein, the audio dataset includes audio data of the source singer and the target singer and the corresponding singer ID; The feature extraction module is used to perform step S20, which involves extracting features from the source singer's audio data to obtain the content features, fundamental frequency, and volume of the source singer's audio data; and extracting the singer ID of the target singer to obtain the embedding of the target singer. In this embodiment, when extracting features from the source singer's audio data, a pre-downloaded HuBERT model is used to extract corresponding content features and prosodic features. Content features primarily represent the core content information of the singing, such as lyrics and melody, while prosodic features include fundamental frequency and volume, which are dynamic features that determine the expressiveness of the singing. Understandably, the HuBERT model is a self-supervised speech pre-training model. By inputting the singer ID and audio data of the source singer from the audio data into the HuBERT model, the extracted content features and prosodic features of the source singer's corresponding audio data can be output. When embedding and extracting the singer ID from the target singer's audio data, an embedding extractor is used to convert the discrete singer ID into continuous vector features for embedding. These continuous vector features are used to represent the unique timbre of the target singer.

[0024] The feature fusion module is used to perform step S30, which fuses the content features, fundamental frequency, volume, and embedding of the source singer with those of the target singer to obtain fused audio feature information. Specifically, the feature fusion module includes a first normalization submodule, a mapping processing submodule, and a fusion submodule, wherein, Please see Figure 2 The first normalization submodule is used to execute step S301, which normalizes the fundamental frequency of the source singer to obtain the normalized fundamental frequency. Understandably, the first normalization submodule is a fundamental frequency fully connected layer. The fundamental frequency fully connected layer uses a preprocessing function to perform logarithmic normalization on the fundamental frequency in order to stabilize the numerical distribution.

[0025] The mapping submodule is used to execute step S302, which projects the source singer's content features, volume, and normalized fundamental frequency onto the same dimension. Understandably, the mapping submodule includes a content feature fully connected layer and a volume fully connected layer. The content feature fully connected layer projects the input content features onto the same dimension; the volume fully connected layer projects the input volume onto the same dimension as the content features, thus eliminating dimensional differences between different features. In one embodiment, the mapping submodule further includes a pitch enhancement offset fully connected layer, used to introduce controllable pitch perturbation information during the training phase to enhance the robustness of the system model.

[0026] The fusion submodule is used to execute step S303, which involves adding and fusing the content features, volume, fundamental frequency, and target singer's embedding obtained after projection processing frame by frame to obtain fused audio feature information. Understandably, the obtained fused audio feature information simultaneously encodes the content, rhythm, and timbre information of the singing. The singing voice conversion output module is used to execute step S40, inputting the fused audio feature information into the trained singing voice conversion model, and outputting the singing voice conversion Mel spectrum and the target singing voice.

[0027] In this embodiment, the singing voice conversion output module is a singing voice conversion model. During the training phase, the fused audio feature information is input into the constructed singing voice conversion model for training and optimization. The loss between the predicted Mel spectrum of the generated audio and the real audio Mel spectrum is calculated, and the loss between them is gradually reduced through iterative training to train the embedding extractor and the singing voice conversion model, finally obtaining the trained embedding extractor and the singing voice conversion model. During the inference phase, the fused audio feature information is input into the trained singing voice conversion model, and the converted singing voice and the corresponding Mel spectrum are output.

[0028] Specifically, the singing voice conversion model includes a Transformer module and a spectrum prediction output module. The Transformer module consists of 12 stacked encoders, each encoder including a multi-head self-attention submodule and a feedforward neural network submodule. Please see Figure 3 The multi-head self-attention submodule is used to execute step S401, which performs multi-head self-attention processing on the fused audio feature information and outputs the self-attention processed audio feature information. Furthermore, the multi-head self-attention submodule includes a second normalization submodule, a multi-head self-attention submodule, and a residual submodule, wherein, Please see Figure 4 The second normalization submodule is used to execute step S4011, which performs layer normalization processing on the fused audio feature information to obtain the fused audio feature information after layer normalization. The multi-head self-attention submodule is used to execute step S4012, which calculates in parallel the correlation weights between all positions in the fused audio feature information after the layer normalization process, and generates a context vector of the fused global context. Understandably, after performing layer normalization processing on the fused audio feature information, the correlation weights of all positions in the sequence are calculated in parallel, which can capture the long-distance contextual dependencies of the singing, thereby generating a context vector of the fused global context.

[0029] The first residual submodule is used to execute step S4013, which involves adding the context vector of the fused global context to the fused audio feature information to obtain the first intermediate fused audio feature information. The second normalization submodule is further configured to perform step S4014, which performs layer normalization processing on the obtained first intermediate fused audio feature information and outputs the self-attention processed audio feature information.

[0030] The feedforward neural network submodule is used to execute step S402, which performs feedforward processing on the self-attention processed audio feature information and outputs encoded audio feature information. Understandably, in this embodiment, the feedforward neural network submodule further performs nonlinear mapping and dimensionality reduction on the aforementioned self-attention processed audio feature information to enhance the model's feature representation capability.

[0031] Furthermore, the feedforward neural network submodule includes a first fully connected layer, a second fully connected layer, a residual layer, and a normalization layer, wherein, Please see Figure 5 The first fully connected layer is used to execute step S4021, which performs nonlinear mapping on the self-attention processed audio feature information to obtain the mapped audio feature information. The second fully connected layer is used to execute step S4022, which performs projection dimension transformation on the audio feature information after mapping transformation to obtain standard dimension audio feature information. The residual layer is used to perform step S4023, which involves adding the residuals of the standard dimension audio feature information and the self-attention processed audio feature information to obtain the second intermediate fused audio feature information. The normalization layer is used to perform step S4024, which performs layer normalization processing on the obtained second intermediate fused audio feature information and outputs the encoded audio feature information.

[0032] The feedforward neural network submodule is also used to execute step S403, inputting the encoded audio feature information into the multi-head self-attention submodule of the next encoder, and repeating steps S401 to S403 until all encoder processing is completed to obtain the final encoded audio feature information.

[0033] The spectrum prediction output module is used to execute step S404, which maps and outputs the final encoded audio feature information to obtain the Mel spectrum of the converted singing.

[0034] Understandably, through the above processing by the Transformer module and the spectrum prediction output module, the final encoded audio feature information encodes the deep features of the global context information, and its shape is consistent with the fused audio features at the input, preserving all information of content, rhythm and timbre.

[0035] The singing voice conversion method based on multi-feature fusion provided by this invention inputs the extracted content features, fundamental frequency, volume of the source singer, and the ID vector features of the target singer into a trained singing voice conversion model. The model generates the embedding of the target singer through an embedding extractor, and outputs the Mel spectrum of the target singer's timbre through feature fusion, Transformer sequence modeling, and spectrum prediction, ultimately restoring the audio to obtain the converted singing voice of "source singer + target singer timbre". This invention explicitly integrates dimensional features such as content features, prosodic features, and identity features (the timbre embedded by the singer), making the generated singing voice more expressive; at the same time, it effectively captures the temporal dependence of the singing voice by combining the long sequence modeling capability of Transformer, improving the accuracy and stability of the Mel spectrum output, and significantly improving the naturalness and fidelity of the generated singing voice; moreover, during singing voice conversion, flexible control of the target singer's timbre is achieved through singer ID embedding, adapting to conversion scenarios with multiple singers, arbitrary source singers, and target singers.

[0036] Understandably, in this embodiment, before using the trained singing conversion model to perform singing conversion, an initial singing conversion model is first constructed, and then the initial singing conversion model is trained and optimized using a dataset containing multi-style singing audio from multiple singers, thereby obtaining a trained singing conversion model.

[0037] Specifically, the training process of a well-trained singing voice conversion model includes: S01, Obtain the singer's audio dataset and preprocess each audio data in the audio dataset to generate a configuration file; Specifically, when training the singing conversion model, a dataset of audio recordings from multiple singers in various styles is first acquired. The directory structure of this dataset contains a file corresponding to each singer's ID, along with a corresponding singing segment. After acquiring the source singers' audio datasets, each audio segment is preprocessed and resampled to 44100Hz to generate a corresponding configuration file. Each singer's file in the generated configuration file contains their own singer ID and singing segment. Understandably, the audio dataset preprocessing process during training is consistent with the audio data preprocessing method described above in the singing conversion method, and will not be elaborated further here. The preprocessed audio dataset is divided into a training set and a test set according to a certain ratio. In this embodiment, a ratio of 0.995:0.005 is preferred. Understandably, the training set is used for model training in the singing conversion model, and in the aforementioned embodiment, the acquired audio dataset can be used for singing conversion using the evaluation set.

[0038] Understandably, in this embodiment, the audio dataset consists of collected audio data from multiple singers and styles. After acquiring the audio dataset, each audio track is preprocessed and resampled to a uniform audio sampling rate of 44100Hz to eliminate format differences. Preferably, the preprocessing method involves slicing each audio track to 3-10 seconds to adapt to the sequence length of the model. Simultaneously, after preprocessing and resampling each audio track to a uniform audio sampling rate, a unique discrete singer ID is generated for each singer, and corresponding configuration files are generated by partitioning file directories according to the singer IDs. Understandably, the generated configuration files contain the singer IDs of all singers, audio segments, and audio paths, etc.

[0039] S02, feature extraction is performed on each audio data in the configuration file to obtain the singer's content features, fundamental frequency, volume, and true Mel spectrum; and the singer's ID is embedded to obtain the singer's embedding; the feature extraction method during training is the same as the feature extraction method in the singing conversion method described above, and will not be described in detail here.

[0040] S03, interpolate the content features of the singer, and use the same starting frame and frame number to slice the base frequency, volume and the interpolated content features to obtain the sliced ​​content features, base frequency and volume; S04. After slicing, the content features, fundamental frequency, volume, and corresponding singer embedding of the singer are fused to obtain the fused audio feature information. The specific training process is the same as the feature fusion implementation steps described above, and will not be repeated here.

[0041] S05, input the fused audio feature information into the Transformer module, and output the final encoded audio feature information; the specific training process is the same as the Transformer module implementation steps described above, and will not be repeated here.

[0042] S06, the final encoded audio feature information is input to the spectrum prediction output module for mapping output to obtain the predicted Mel spectrum, and the trained singing conversion model is determined based on the loss between the predicted Mel spectrum and the real Mel spectrum.

[0043] Specifically, during training, the fused audio feature information of the source singer is used as the input to the vocal conversion model, and the corresponding predicted Mel spectrum is used as the training target to perform supervised training on the vocal conversion model. The difference between the predicted Mel spectrum and the true Mel spectrum is used as the loss. The model parameters (including the weights of the Transformer, the spectrum prediction output module, and the embedding extractor) are iteratively updated through a gradient descent-based backpropagation algorithm and an optimizer. The loss function is used to calculate the loss between the predicted target Mel spectrum and the true Mel spectrum until the vocal conversion model converges, thus obtaining a trained vocal conversion model that minimizes the loss difference between the predicted Mel spectrum and the true Mel spectrum. Furthermore, it can be understood that during model training, the weights of the embedding extractor are constantly being updated and changed. Therefore, the extracted singer embeddings are also constantly being updated and changed until the model training ends and the trained vocal conversion model is obtained. Only then are the weights of the embedding extractor and the singer embeddings fixed.

[0044] The singing voice conversion method based on multi-feature fusion provided by this invention, in the training phase, extracts multi-dimensional features such as HuBERT, fundamental frequency, and volume from each audio file in the training set, and extracts the embedding of the corresponding singer ID through an embedding extractor. The obtained features are then fused. The fused features are fed into the singing voice conversion model for training. By calculating the loss between the Mel spectrum of the generated audio and the Mel spectrum of the original audio, and gradually reducing this loss through iterative training, the embedding extractor and singing voice conversion model are trained, ultimately resulting in a well-trained embedding extractor and singing voice conversion model. In the inference phase, by using the trained embedding extractor and singing voice conversion model, only the embedded singer ID needs to be replaced to transfer the timbre of the target singer to any audio file in the audio dataset of multiple source singers, generating audio containing the given target singer timbre and the audio content of the source singers. This method can achieve high-quality and high-stability generation of singing voice acoustic features, effectively improving the naturalness and expressiveness of singing voice conversion.

[0045] It should be understood that, although Figure 1-5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-5At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0046] Based on the same inventive concept, please refer to Figure 6 This application also provides a vocal conversion device for implementing the methods described above. The solution provided by this device is similar to the implementation described in the methods above; therefore, the specific limitations of the vocal conversion device provided below can be found in the limitations of the vocal conversion methods described above, and will not be repeated here. Each module in the vocal conversion device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0047] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a singing voice conversion method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0048] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0049] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the singing voice conversion method in the above-described method embodiments.

[0050] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the singing voice conversion method in the above-described method embodiments.

[0051] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0052] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A singing voice conversion method based on multi-feature fusion, characterized in that, include: S10, Obtain the audio dataset; wherein, the audio dataset includes the audio data of the source singer and the target singer, and the corresponding singer ID; S20, feature extraction is performed on the audio data of the source singer to obtain the content features, fundamental frequency, and volume of the source singer's audio data; embedding extraction is performed on the singer ID of the target singer to obtain the embedding of the target singer; S30, feature fusion is performed on the content features, fundamental frequency, volume of the source singer and the embedding of the target singer to obtain fused audio feature information; S40, the fused audio feature information is input into the trained singing conversion model, and the Mel spectrum after singing conversion is output.

2. The singing voice conversion method according to claim 1, characterized in that, S30 includes: S301, normalize the fundamental frequency of the source singer to obtain the normalized fundamental frequency; S302, Project the content features, volume and normalized fundamental frequency of the source singer, and map the content features, volume and normalized fundamental frequency of the source singer to the same dimension. S303, the content features, volume, fundamental frequency of the source singer and the embedding of the target singer obtained after projection processing are added and fused frame by frame to obtain fused audio feature information.

3. The singing voice conversion method according to claim 1, characterized in that, S40 includes: S401, Perform multi-head self-attention processing on the fused audio feature information, and output the self-attention processed audio feature information; S402, perform feedforward processing on the audio feature information after self-attention processing, and output the encoded audio feature information; S403, input the encoded audio feature information into the multi-head self-attention submodule of the next encoder, and repeat steps S401 to S403 until all encoder processing is completed to obtain the final encoded audio feature information; S404, the final encoded audio feature information is mapped and output to obtain the Mel spectrum after the singing voice is converted.

4. The singing voice conversion method according to claim 3, characterized in that, S401 includes: S4011, The fused audio feature information is subjected to layer normalization processing to obtain the fused audio feature information after layer normalization processing; S4012, Parallel calculation of the correlation weights between all positions in the fused audio feature information after the layer normalization process, generating a context vector of the fused global context; S4013, the context vector of the fused global context is added to the fused audio feature information by residual addition to obtain the first intermediate fused audio feature information; S4014, the obtained first intermediate fused audio feature information is subjected to layer normalization processing, and the self-attention processed audio feature information is output.

5. The singing voice conversion method according to claim 3, characterized in that, S402 includes: S4021, Nonlinear mapping is performed on the audio feature information after self-attention processing to obtain the mapped audio feature information; S4022, Perform projection dimension transformation on the mapped audio feature information to obtain standard dimension audio feature information; S4023, The residuals of the standard dimension audio feature information and the self-attention processed audio feature information are added to obtain the second intermediate fused audio feature information; S4024, the obtained second intermediate fused audio feature information is subjected to layer normalization processing, and the encoded audio feature information is output.

6. The singing voice conversion method according to claim 1, characterized in that, In S40, the training process of the singing voice conversion model includes: S01, Obtain the singer's audio dataset and preprocess each audio data in the audio dataset to generate a configuration file; S02, perform feature extraction on each audio data in the configuration file to obtain the singer's content features, fundamental frequency, volume, and true Mel spectrum; and perform embedding extraction on the singer's singer ID to obtain the singer's embedding. S03, interpolate the content features of the singer, and use the same starting frame and frame number to slice the base frequency, volume and the interpolated content features to obtain the sliced ​​content features, base frequency and volume; S04, input the content features, fundamental frequency and volume after the slice processing to the feature fusion module, perform feature fusion on the content features, fundamental frequency and volume of the singer and the corresponding singer embedding, and obtain the fused audio feature information; S05, input the fused audio feature information into the Transformer module, and output the final encoded audio feature information; S06, the final encoded audio feature information is input to the spectrum prediction output module for mapping output to obtain the predicted Mel spectrum, and the trained singing conversion model is determined based on the loss between the predicted Mel spectrum and the real Mel spectrum.

7. A singing voice conversion device with multi-feature fusion, characterized in that, include: The module includes a data acquisition module, a feature extraction module, a feature fusion module, and a vocal conversion output module. The data acquisition module is used to acquire an audio dataset; wherein, the audio dataset includes audio data of the source singer and the target singer, and the corresponding singer ID; The feature extraction module is used to extract features from the audio data of the source singer to obtain the content features, fundamental frequency, and volume of the source singer's audio data; and to extract the singer ID of the target singer to obtain the embedding of the target singer. The feature fusion module is used to perform feature fusion on the content features, fundamental frequency, volume of the source singer and the embedding of the target singer to obtain fused audio feature information. The singing voice conversion output module is used to input the fused audio feature information into the trained singing voice conversion model and output the singing voice conversion Mel spectrum.

8. The singing voice conversion device according to claim 7, characterized in that, The feature extraction module is further used to extract the embedding of the source singer ID to obtain the embedding of the source singer; the feature fusion module is further used to fuse the content features, fundamental frequency, volume and embedding of the source singer to obtain fused audio feature information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the singing voice conversion method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the singing voice conversion method according to any one of claims 1 to 6.

Images