Song feature conversion method, electronic device, storage medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a song feature conversion method, an electronic device and a storage medium. The song feature conversion method provided by the application needs to acquire a source audio and a target sound identifier first, and then input the source audio and the target sound identifier into a pre-trained audio processing model. The audio processing model comprises a vocoder, a music emotion recognition network and a variational adversarial network. The source audio is subjected to acoustic feature extraction based on the vocoder to obtain content feature parameters. The source audio is subjected to emotion information extraction based on the music emotion recognition network to obtain a source emotion feature vector. Further, the source emotion feature vector is replaced with a target emotion feature vector through the variational adversarial network based on the target sound identifier. Finally, the target emotion feature vector and the content feature parameters are subjected to reconstruction processing based on the variational adversarial network to obtain a target audio after song feature conversion, so that the dependence on parallel data can be eliminated in the process of realizing song feature conversion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a singing voice feature conversion method, an electronic device and a storage medium. BACKGROUND

[0002] Voice conversion (VC) refers to converting non-linguistic features of a voice while retaining linguistic features of the voice. The linguistic features of the voice refer to the content of the voice corresponding to the pronunciation of a language, and the non-linguistic features of the voice include an accent, a timbre, a speaking style and the like. Singing voice conversion (SVC) is a technology of converting non-linguistic features of singing of a song from a source singer to a target singer without changing the language content of the singing.

[0003] In the related art, singing voice conversion requires paired parallel data, that is, the same lyrics are sung by different people to obtain parallel data. However, such data is difficult to obtain in real-world applications. Therefore, how to eliminate the dependence on parallel data in the process of implementing singing voice conversion has become a problem to be solved in the industry. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a singing voice feature conversion method, an electronic device and a storage medium, which can eliminate the dependence on parallel data in the process of implementing singing voice conversion.

[0005] The singing voice feature conversion method according to the first aspect of the present application comprises:

[0006] obtaining a source audio and a target voice identifier, the source audio comprising an audio singing section of a source singer;

[0007] inputting the source audio and the target voice identifier into a pre-trained audio processing model, the audio processing model comprising a vocoder, a music emotion recognition network and a variational autoencoder network;

[0008] performing acoustic feature extraction on the source audio based on the vocoder to obtain content feature parameters;

[0009] performing emotion information extraction on the source audio based on the music emotion recognition network to obtain a source emotion feature vector;

[0010] replacing the source emotion feature vector with a target emotion feature vector based on the target voice identifier through the variational autoencoder network;

[0011] Reconstruct the target audio after singing feature conversion based on the variational adversarial network on the target emotional feature vector and the content feature parameter.

[0012] According to some embodiments of the present application, before the source audio and the target sound identifier are input into the pre-trained audio processing model, the method further comprises pre-training the audio processing model, specifically comprising:

[0013] Obtain a training data set, which includes a first number of training audios, each of which is configured with a corresponding training sound identifier;

[0014] Group the first number of training audios to obtain a second number of training audio groups, each of which includes training audios sung by different singers for the same audio segment;

[0015] Input the second number of training audio groups into the original audio processing model, and iteratively train the audio processing model based on the training audios and the training sound identifiers;

[0016] When the audio processing model meets the first preset condition in the iterative training, the pre-trained audio processing model is obtained.

[0017] According to some embodiments of the present application, inputting the second number of training audio groups into the original audio processing model and iteratively training the audio processing model based on the training audios and the training sound identifiers comprises:

[0018] In each round of iterative training, an iteration audio group is selected from the second number of training audio groups, and a training source audio and a training target audio are selected from the iteration audio group;

[0019] Extract the acoustic features of the training source audio based on the vocoder to obtain training content parameters;

[0020] Extract the emotional information of the training source audio based on the pre-trained music emotion recognition network to obtain an emotional initial vector;

[0021] Extract the emotional information of the training target audio based on the pre-trained music emotion recognition network to obtain an emotional replacement vector;

[0022] Determine the training sound identifier configured for the training target audio as a sound replacement identifier, and replace the emotional initial vector with the emotional replacement vector through the variational adversarial network based on the sound replacement identifier;

[0023] reconstructing the emotion replacement vector and the training content parameter based on the variational adversarial network to obtain predicted audio;

[0024] performing loss calculation on the training source audio, the training target audio and the predicted audio to obtain predicted deviation data;

[0025] updating model parameters of the audio processing model based on the predicted deviation data after each round of the iterative training.

[0026] According to some embodiments of the present application, the training content parameter includes a training prosody parameter and a training spectrum parameter, and the reconstructing the emotion replacement vector and the training content parameter based on the variational adversarial network to obtain predicted audio includes:

[0027] performing language feature analysis on the training spectrum parameter to obtain a training hidden vector, the training hidden vector being used to represent sentence content of the training source audio;

[0028] integrating and splicing the emotion replacement vector, the training prosody parameter and the training hidden vector to obtain the predicted audio.

[0029] According to some embodiments of the present application, the predicted deviation data includes a loss function value, and the performing loss calculation on the training source audio, the training target audio and the predicted audio to obtain predicted deviation data includes:

[0030] performing first loss calculation on the training source audio and the training hidden vector to obtain an encoding loss element corresponding to the music emotion recognition network;

[0031] performing second loss calculation on the training source audio, the training target audio and the predicted audio to obtain a decoding loss element corresponding to the variational adversarial network;

[0032] performing loss construction based on the encoding loss element and the decoding loss element to obtain a predicted loss function;

[0033] performing loss calculation on the training source audio, the training target audio and the predicted audio based on the predicted loss function to obtain the loss function value.

[0034] According to some embodiments of the present application, before the performing loss calculation on the training source audio, the training target audio and the predicted audio based on the predicted loss function to obtain the loss function value, the method further includes:

[0035] performing third loss calculation on the training source audio, the training target audio and the predicted audio to obtain a discrimination loss element corresponding to the variational adversarial network;

[0036] integrate the discriminative loss elements into the prediction loss function based on preset weight factors.

[0037] According to some embodiments of the present application, the acoustic feature extraction of the vocoder on the source audio includes:

[0038] The source prosody feature parameter is used to represent the prosody content of the source audio.

[0039] The frequency spectrum feature extraction of the vocoder on the source audio includes obtaining a frequency spectrum feature parameter.

[0040] The source prosody feature parameter and the frequency spectrum feature parameter are integrated to obtain the content feature parameter.

[0041] According to some embodiments of the present application, the reconstruction processing of the variational autoencoder on the target emotion feature vector and the content feature parameter includes:

[0042] The frequency spectrum feature parameter and the source prosody feature parameter are obtained from the content feature parameter.

[0043] The language feature analysis on the frequency spectrum feature parameter includes obtaining a sentence latent vector, which is used to represent the sentence content of the source audio.

[0044] The target audio is obtained by integrating and splicing the target emotion feature vector, the source prosody feature parameter and the sentence latent vector.

[0045] In a second aspect, the embodiments of the present application provide an electronic device, including a memory and a processor, the memory stores a computer program, and the processor implements the song feature conversion method according to any one of the embodiments of the first aspect of the present application when executing the computer program.

[0046] In a third aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the song feature conversion method according to any one of the embodiments of the first aspect of the present application.

[0047] According to the song feature conversion method, the electronic device and the storage medium, at least the following beneficial effects are achieved:

[0048] The song feature conversion method of the present application needs to obtain a source audio and a target sound identifier first, the source audio includes an audio singing section of a source singer, and then the source audio and the target sound identifier are input into a pre-trained audio processing model, the audio processing model includes a vocoder, a music emotion recognition network and a variational adversarial network, acoustic feature extraction is performed on the source audio based on the vocoder to obtain content feature parameters, further, emotion information extraction is performed on the source audio based on the music emotion recognition network to obtain a source emotion feature vector, and then further, the source emotion feature vector is replaced with a target emotion feature vector through the variational adversarial network based on the target sound identifier, and finally, the target emotion feature vector and the content feature parameters are reconstructed based on the variational adversarial network to obtain the target audio after the song feature conversion. It should be pointed out that the song feature conversion is realized through the audio processing model of the present application, which can realize the song feature conversion without relying on parallel data.

[0049] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings.

[0051] Figure 1 An optional flowchart of a song feature conversion method provided by an embodiment of the present application;

[0052] Figure 2 An optional structure schematic block diagram of a music emotion recognition network of an embodiment of the present application;

[0053] Figure 3 Another optional flowchart of a song feature conversion method provided by an embodiment of the present application;

[0054] Figure 4 An optional flowchart of step S303 in the method; Figure 3

[0055] An optional structure schematic block diagram of an audio processing model of an embodiment of the present application; Figure 5

[0056] An optional flowchart of step S406 in the method; Figure 6 Figure 4 An optional flowchart of step S407 in the method;

[0057] Figure 7 Figure 4 An optional flowchart of step S407 in the method;

[0058] Figure 8 ​​Another optional flowchart of a singing voice feature conversion method provided by an embodiment of the present application is shown in FIG. 6.

[0059] Figure 9 FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] Embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, in which the same or similar notations are used to denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only, and are merely used to explain the present application, and cannot be understood as limiting the present application.

[0061] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If it is described as first, second, etc., it is only used to distinguish technical features for the purpose, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implying the number of the indicated technical features or the order of the indicated technical features.

[0062] In the description of the present application, it should be understood that the description of the position, such as up, down, left, right, front, back, etc. is based on the position or location relationship shown in the drawings, which is only for the purpose of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present application.

[0063] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present description, the exemplary description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0064] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution. In addition, the identification of the specific steps in the following does not represent the limitation of the order and execution logic of the steps, and the execution order and execution logic between the steps should be understood and inferred with reference to the content described in the embodiments.

[0065] Voice Conversion (VC) refers to converting non-linguistic features of a voice while preserving linguistic features of the voice. The linguistic features of the voice refer to the content of the voice corresponding to the pronunciation of a language, and the non-linguistic features of the voice include an accent, a timbre, a speaking style, and the like. Singing Voice Conversion (SVC) is a technology of converting non-linguistic features of singing of a song from a source singer to a target singer without changing the language content of the singing.

[0066] In the related art, singing voice conversion requires paired parallel data, that is, the same lyrics are sung by different people to obtain parallel data. However, such data is difficult to obtain for real-world applications. Therefore, how to achieve singing voice conversion without relying on parallel data has become a problem to be solved in the industry. In addition, the current singing voice conversion task does not take into account the emotion factor, and some exemplary embodiments of the present application model and integrate the singer-related characteristics into singing voice conversion.

[0067] The present application aims to at least solve one of the technical problems in the prior art. To this end, the present application provides a singing voice conversion method, an electronic device, and a storage medium, which can achieve singing voice conversion without relying on parallel data.

[0068] The singing voice conversion method will be further described below with reference to the accompanying drawings.

[0069] Reference Figure 1 The singing voice conversion method of the first aspect of the present application shown can include, but is not limited to, the following steps S101 to S106.

[0070] Step S101, obtaining a source audio and a target voice identifier, the source audio including an audio singing section of a source singer;

[0071] Step S102, inputting the source audio and the target voice identifier into a pre-trained audio processing model, the audio processing model including a vocoder, a music emotion recognition network, and a variational autoencoder network;

[0072] Step S103, performing acoustic feature extraction on the source audio based on the vocoder to obtain content feature parameters;

[0073] Step S104, performing emotion information extraction on the source audio based on the music emotion recognition network to obtain a source emotion feature vector;

[0074] Step S105, replacing the source emotion feature vector with a target emotion feature vector based on the target voice identifier through the variational autoencoder network;

[0075] Step S106, the target audio after the singing feature conversion is obtained by reconstructing the target emotional feature vector and the content feature parameter based on the variational adversarial network.

[0076] The singing feature conversion method shown in steps S101 to S106 needs to obtain the source audio and the target sound identifier first, the source audio includes the audio singing section of the source singer, and then the source audio and the target sound identifier are input into the pre-trained audio processing model, the audio processing model includes a vocoder, a music emotion recognition network, and a variational adversarial network. The acoustic feature of the source audio is extracted based on the vocoder to obtain the content feature parameter. Further, the emotional information of the source audio is extracted based on the music emotion recognition network to obtain the source emotional feature vector. Further, the source emotional feature vector is replaced by the target emotional feature vector through the variational adversarial network based on the target sound identifier. Finally, the target audio after the singing feature conversion is obtained by reconstructing the target emotional feature vector and the content feature parameter based on the variational adversarial network. It should be pointed out that the singing feature conversion is realized by the audio processing model of the present application, which can realize the singing feature conversion without relying on parallel data.

[0077] In step S101 of some embodiments, the source audio and the target sound identifier are obtained, and the source audio includes the audio singing section of the source singer. It should be noted that the source audio refers to the audio singing section of the source singer, and the target sound identifier is used to identify the target singer. In some exemplary embodiments, the process of singing feature conversion is actually aimed at the audio singing section of the source singer in the source audio, and the singing feature of the source singer is converted to the singing feature of the target singer. It should be noted that the target sound identifier is used as a reference for singing feature conversion when the singing feature conversion is performed, and the audio processing model is used to convert the singing feature of the source singer to the singing feature of the target singer based on the target sound identifier. It should be understood that converting the singing feature of the source singer to the singing feature of the target singer can convert the emotional feature of the source singer to the emotional feature of the target singer, or convert part of the content feature of the source singer to the content feature of the target singer.

[0078] It should be understood that sound mainly includes three aspects of characteristics of loudness, pitch and timbre. Among them, loudness refers to the size of the sound that a person subjectively feels, which is determined by the amplitude and the distance of the person from the sound source. The greater the amplitude, the greater the loudness, and the smaller the distance between the person and the sound source, the greater the loudness. The sound characteristics in this application specifically refer to the attribute characteristics of audio, so the distance between the person and the sound source is not considered here, and the amplitude of the audio is regarded as the characteristic of loudness. The pitch is determined by the frequency (the number of vibrations completed in a unit of time). The faster the object vibrates, the higher the pitch of the sound emitted, and the slower the vibration, the lower the pitch of the sound emitted. The timbre, also known as tone, is determined by the waveform of the sound. The timbre itself is an abstract thing, but the waveform corresponds to the intuitive representation of this abstraction. Different waveforms correspond to different timbres.

[0079] It needs to be clear that the singing characteristics are a special kind of sound characteristics. In addition to analyzing the amplitude, frequency and waveform of the sound, the analysis of some characteristic parameters is also included in the analysis of the singer characteristics. These characteristic parameters can include, but are not limited to:

[0080] The fundamental frequency F0 is used to reflect a series of pulse signals emitted by the vocal cords. If this part of the pulse signal is divided into periodic signals and aperiodic signals, the prosody information and structural information of the singing can be obtained.

[0081] The spectrum parameter (SP) is also called the spectrum envelope. After the human voice is emitted by the vocal cords, through the filter sound channel, that is, through the common modulation of the throat cavity, oral cavity, lips, teeth and other channel parts, humans can emit different timbres and different vowels and consonants. It should be pointed out that the throat cavity, oral cavity, lips, teeth and other channel parts together form a filter system, which describes the timbre and content of the singing voice, and is the frequency response of a linear time-invariant system.

[0082] In step S102 of some embodiments, the source audio and the target sound identification are input into the pre-trained audio processing model. The audio processing model includes a vocoder, a music emotion recognition network and a variational adversarial network. It should be noted that the audio processing model is pre-trained, and in the pre-training process, the audio processing model gradually acquires the conversion capability of converting the singing characteristics of the source singer to the singing characteristics of the target singer. The audio processing model can specifically include a vocoder, a music emotion recognition network and a variational adversarial network.

[0083] In step S103 of some embodiments, the source audio is subjected to acoustic feature extraction based on a vocoder to obtain content feature parameters. It should be noted that a vocoder is a type of speech codec that analyzes and synthesizes human voice signals for audio data compression, multiplexing, voice encryption or speech transformation; and the content feature parameters refer to feature parameters reflecting the prosody information, structure information, timbre, speech content and other content information of the singing voice, such as the above-mentioned fundamental frequency F0 and spectral parameters. It should be pointed out that the vocoder can analyze the speech signal in the source audio, extract the feature parameters of the speech signal for encoding and encryption. The analysis process of the vocoder can be carried out in the frequency domain, and the speech signal is subjected to frequency spectrum analysis, the voiced and unvoiced sounds are identified, the voiced fundamental frequency is determined, and then the unvoiced-voiced judgment, the voiced fundamental frequency and the spectral envelope are selected as the feature parameters for transmission. The analysis process of the vocoder can also be carried out in the time domain, and some parameters are extracted for linear prediction or correlation analysis of the speech signal according to the working principle. According to the working principle, the vocoder can be divided into: channel vocoder, formant vocoder, pattern vocoder, linear prediction vocoder, correlation vocoder and orthogonal function vocoder.

[0084] It should be noted that the current common vocoder is divided into two types, one is based on traditional signal processing algorithm, such as STRAIGHT, Griffim Lim, WORL D; and the second is the vocoder based on neural network (MelGAN, LPCNet, etc.) that has emerged in recent years. Compared with the vocoder based on neural network, the calculation complexity of the WORL D vocoder is lower, which is more suitable for low-performance computing devices, and the WORL D vocoder does not need a large amount of data for training and can be directly calculated based on the signal processing algorithm. It should be emphasized that the WORL D vocoder can realize real-time and high-quality speech synthesis. Therefore, in some more preferred embodiments, the WORL D vocoder can be selected as the vocoder of the present application and applied to the audio processing model.

[0085] According to some embodiments of the present application, in step S103, the source audio can be subjected to fundamental frequency feature extraction based on a vocoder to obtain source prosody feature parameters, and then subjected to spectral feature extraction based on the vocoder to obtain spectral feature parameters, and finally the source prosody feature parameters and the spectral feature parameters are integrated to obtain the content feature parameters.

[0086] In step S104 of some embodiments, the source audio is subjected to emotion information extraction based on a music emotion recognition network to obtain a source emotion feature vector. It should be noted that the music emotion recognition network refers to a neural network model for extracting emotion features from singing. Music emotion recognition (MER) is a process of using a computer to extract and analyze music features, forming a mapping relationship between music features and emotion space, and identifying the emotions expressed by music. Music features are usually extracted from audio signals, symbolic scores, lyrics texts, or even biological characteristics such as electroencephalograms. Emotion space can be represented by a limited number of discrete categories or an infinite number of points in a continuous multidimensional space. Music emotion recognition belongs to the interdisciplinary research field of music psychology, audio signal processing, and natural language processing, and music emotion recognition is a subtask of music information retrieval. Music emotion recognition can be widely applied in the fields of music recommendation, retrieval, visualization, automatic composition, psychological treatment, etc.

[0087] With reference to Figure 2 In some more specific embodiments of the present application, the music emotion recognition network can specifically include a convolutional layer, a bidirectional long short-term memory layer, and an attention layer. It should be noted that a traditional convolutional neural network (CNN) learns spatial information within a receptive field, and as the depth of learning and training increases, the features learned by the convolutional neural network become more and more abstract. For music emotion recognition, the convolutional neural network needs to learn abstract spatial features, so the convolutional layer is set after the convolutional layer. It should be noted that the bidirectional long short-term memory layer is specifically a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory) for capturing long-distance dependency information of context in abstract spatial features, and then extracting features with weight distribution in time domain through the attention mechanism of the attention layer, i.e., obtaining an emotion feature vector for representing singing emotion features.

[0088] It should be noted that the same song can be sung by different singers to produce different emotional effects. These emotional effects can be happy and joyful, depressed and lost, carefree and unrestrained, or other types of feelings in human perception. For these emotional effects, the above-mentioned emotion feature vector needs to be extracted from the singing, and based on these emotion feature vectors, the emotional effects contained in the singing can be quantitatively analyzed.

[0089] In step S105 of some embodiments, the source emotional feature vector is replaced by the target emotional feature vector by the variational adversarial network based on the target sound identification. In some example embodiments of the present application, the variational adversarial network (VAW-GAN) refers to a generative adversarial network (GAN) based on a variational autoencoder (VAE). It should be emphasized that, in view of the abstractness of emotional features, the related art generally does not consider the factor of emotion when performing a singing voice feature conversion task, while the variational adversarial network of the present application is used to replace the source emotional feature vector by the target emotional feature vector, aiming to incorporate emotional feature conversion into the process of singing voice feature conversion and improve the smoothness of singing voice feature conversion.

[0090] In step S106 of some embodiments, the target emotional feature vector and the content feature parameter are reconstructed by the variational adversarial network to obtain the target audio after singing voice feature conversion. It should be noted that the variational adversarial network is also used to reconstruct the target emotional feature vector and the content feature parameter to obtain the target audio after singing voice feature conversion. It should be understood that after the acoustic feature extraction of the source audio to obtain the content feature parameter, and the replacement of the source emotional feature vector by the target emotional feature vector by the variational adversarial network, the target emotional feature vector and the content feature parameter need to be further reconstructed to replace the emotional feature vector while retaining the content information such as prosody information, structural information, timbre, and speech content in the source audio, so as to complete the overall process of singing voice feature conversion.

[0091] According to some embodiments of the present application, in step S106, the spectral feature parameter and the source prosody feature parameter can be obtained from the content feature parameter, and the spectral feature parameter is analyzed for language features to obtain a sentence hidden vector, which is used to represent the sentence content of the source audio. Further, the target emotional feature vector, the source prosody feature parameter, and the sentence hidden vector are integrated and spliced to obtain the target audio.

[0092] Reference Figure 3 According to some embodiments of the present application, before step S102, the singing voice feature conversion method further includes pre-training an audio processing model, which can specifically include, but is not limited to, the following steps S301 to S304.

[0093] In step S301, a training data set is obtained, which includes a first number of training audios, each of which is configured with a one-to-one corresponding training sound identification;

[0094] Step S302, the first number of training audios is grouped to obtain a second number of training audio groups, each training audio group including training audios of different singers singing the same audio piece;

[0095] Step S303, the second number of training audio groups is input into the original audio processing model, and the audio processing model is iteratively trained based on the training audios and the training sound identifiers;

[0096] Step S304, when the audio processing model meets a first preset condition in the iterative training, a pre-trained audio processing model is obtained.

[0097] In step S301 of some embodiments, a training data set is obtained. It should be noted that the training data set includes a first number of training audios, each training audio being configured with a one-to-one training sound identifier, wherein the training sound identifier is used to identify which singer the training audio specifically corresponds to. According to some embodiments provided by the present application, the training data set can use some commonly used corpora, such as the DAMP corpus and the NUS-48E corpus, both of which contain songs sung by different singers. In some more specific embodiments, the NUS-48E corpus can be used, and 48 songs sung by 12 professional singers can be selected as the training data set.

[0098] Step S302, the first number of training audios is grouped to obtain a second number of training audio groups, each training audio group including training audios of different singers singing the same audio piece. It should be noted that the first number of training audios can be grouped based on training audios of the same audio piece, and in each training audio group, training audios of different singers singing the same audio piece are grouped.

[0099] Step S303, the second number of training audio groups is input into the original audio processing model, and the audio processing model is iteratively trained based on the training audios and the training sound identifiers. It should be emphasized that the audio processing model gradually acquires the conversion capability of converting the singing features of the source singer to the singing features of the target singer in the pre-training process. It should be noted that the audio processing model needs to be iteratively trained according to the training audio groups, because different training audio groups correspond to one piece of singing, and if training samples of different audio pieces are used to train the audio processing model, it is difficult to improve the ability of the audio processing model to convert singing features for the same piece of singing.

[0100] Reference Figure 4 According to some embodiments of the present application, step S303 can include, but is not limited to, steps S401 to S408.

[0101] Step S401, in each round of iterative training, an iteration audio group is selected from the second number of training audio groups, and a training source audio and a training target audio are selected from the iteration audio group;

[0102] Step S402, acoustic feature extraction is performed on the training source audio based on a vocoder, to obtain a training content parameter;

[0103] Step S403, emotion information extraction is performed on the training source audio based on a pre-trained music emotion recognition network, to obtain an emotion initial vector;

[0104] Step S404, emotion information extraction is performed on the training target audio based on the pre-trained music emotion recognition network, to obtain an emotion replacement vector;

[0105] Step S405, a training sound identifier configured for the training target audio is determined as a sound replacement identifier, and the emotion initial vector is replaced by the emotion replacement vector based on the sound replacement identifier through a variational autoencoder network;

[0106] Step S406, the emotion replacement vector and the training content parameter are reconstructed based on the variational autoencoder network, to obtain a predicted audio;

[0107] Step S407, loss calculation is performed based on the training source audio, the training target audio and the predicted audio, to obtain predicted deviation data;

[0108] Step S408, after each round of iterative training, model parameters of an audio processing model are updated based on the predicted deviation data.

[0109] In step S401 of some embodiments, in each round of iterative training, an iteration audio group is selected from the second number of training audio groups, and a training source audio and a training target audio are selected from the iteration audio group. It should be noted that, in one iteration round, an audio file to be replaced as emotion features is referred to as a training source audio, and an audio file to be replaced as emotion features is referred to as a training target audio. It should be emphasized that, since the training source audio and the training target audio are selected from the same iteration audio group, the training source audio and the training target audio are actually singing audios of two singers for the same audio.

[0110] In step S402 of some embodiments, the vocoder is used to perform acoustic feature extraction on the training source audio to obtain training content parameters. It should be noted that the training content parameters are used to represent the feature parameters of the vocal content of the training audio, such as the fundamental frequency F0 and the spectral parameters of the training sample. It should be emphasized that the purpose of iterative training is to enable the audio processing model to gradually acquire the conversion capability of converting the singing features of the source singer to the singing features of the target singer in the pre-training process. In some exemplary embodiments, the singing features of the source singer are converted to the singing features of the target singer, which can specifically be the conversion of the emotional features of the source singer to the emotional features of the target singer, and in this process, the content features of the audio can remain unchanged. Therefore, in order to realize the training of the audio processing model to achieve this capability, acoustic feature extraction on the training source audio based on the vocoder is required before the emotional features of the training audio are extracted.

[0111] In steps S403-S404, the pre-trained music emotion recognition network is used to perform emotion information extraction on the training source audio to obtain an emotion initial vector, and the pre-trained music emotion recognition network is used to perform emotion information extraction on the training target audio to obtain an emotion replacement vector. It should be noted that the emotion initial vector is used to represent the emotional features of the training source audio, and the emotion replacement vector is used to represent the emotional features of the training target audio. It should be pointed out that the music emotion recognition network has undergone pre-training of music emotion feature extraction capability before being used for iterative training of the audio processing model. The pre-training of the music emotion recognition network is mainly performed by relying on the labeled audio dataset for supervised training. It should be clear that the pre-training of the music emotion recognition network aims to enable the music emotion recognition network to have emotion feature extraction capability, or to improve the emotion feature extraction capability of the music emotion recognition network.

[0112] In step S405, the training sound identifier configured for the training target audio is determined as a sound replacement identifier, and the emotion initial vector is replaced by the emotion replacement vector through the variational autoencoder based on the sound replacement identifier. It should be emphasized that each training audio is configured with a one-to-one corresponding training sound identifier, so the training sound identifier can be used as a lookup index for the training audio. The sound replacement identifier refers to the training sound identifier configured for the training target audio, and the emotion initial vector is replaced by the emotion replacement vector through the variational autoencoder based on the sound replacement identifier, which means that the emotion initial vector is replaced by the emotion replacement vector corresponding to the sound replacement identifier through the variational autoencoder with the sound replacement identifier as the lookup index. In this way, the singing feature conversion in the current iteration round can be realized.

[0113] Step S406, the variational autoencoder is used to reconstruct the emotion replacement vector and the training content parameter to obtain the predicted audio. It should be noted that the variational autoencoder is used to reconstruct the emotion replacement vector and the training content parameter, and the predicted audio can be obtained.

[0114] Steps S407 to S408, loss calculation is performed based on the training source audio, the training target audio and the predicted audio to obtain prediction deviation data. After each iteration training, the model parameters of the audio processing model are updated based on the prediction deviation data. It should be noted that after the predicted audio is obtained, loss calculation is performed based on the training source audio, the training target audio and the predicted audio to obtain prediction deviation data for measuring the singing feature conversion accuracy of the audio processing model. After each iteration training, the model parameters of the audio processing model are updated based on the prediction deviation data, so that the audio processing model gradually has the conversion capability of converting the singing features of the source singer to the singing features of the target singer in the iteration training process, until the audio processing model meets the first preset condition in the iteration training, and a pre-trained audio processing model is obtained.

[0115] Through steps S401 to S408, the iteration training of the audio processing model is a process with many iteration rounds. When the iteration rounds are less, the ability of the variational autoencoder to replace the emotion initial vector with the emotion replacement vector is crossed, and the singing feature conversion is difficult to achieve the expected effect. As the iteration rounds gradually increase, the ability of the variational autoencoder to replace the emotion initial vector with the emotion replacement vector is gradually improved, until the audio processing model under the current iteration round has reached the expected pre-training process, and the accuracy of the singing feature conversion has reached the degree that can be applied to the actual scene. At this time, the audio processing model under the current iteration round is output, and a pre-trained audio processing model is obtained, thereby improving the conversion accuracy of the audio processing model in the singing feature conversion, which is a more preferred embodiment.

[0116] Reference Figure 5The illustrated embodiment shows a structural block diagram of an optional audio processing model. It should be noted that after the training source audio input audio processing model is trained, the training source audio is processed by the vocoder and the music emotion recognition network respectively. Among them, the acoustic feature extraction of the training source audio based on the vocoder can obtain the training content parameters, and the training content parameters can specifically include the training prosody parameters and the training spectrum parameters. In addition, the pre-trained music emotion recognition network extracts the emotion information of the training source audio to obtain the emotion initial vector. It should be pointed out that in order to replace the emotion initial vector with the emotion replacement vector, first, the one-to-one corresponding training target audio is found by querying the sound replacement identifier, and then the pre-trained music emotion recognition network is used to extract the emotion information of the training target audio to obtain the emotion replacement vector, so that the emotion initial vector can be replaced with the emotion replacement vector. Further, after obtaining the training prosody parameters and the training spectrum parameters, and replacing the emotion initial vector with the emotion replacement vector, the training prosody parameters, the training spectrum parameters and the emotion replacement vector can be input into the variational autoencoder. It should be noted that the variational autoencoder includes an encoder, a decoder and a discriminator. In some more specific embodiments, the variational autoencoder can include an encoder, a decoder and a discriminator. It should be noted that the encoder is used to convert the input data into a set of encoding vectors, which can be realized by a convolutional neural network, a recurrent neural network or other neural networks. The main function of the encoder is to convert the input data into a low-dimensional dense representation, so as to extract the important features of the data. In some embodiments, the output of the encoder is also called "hidden vector" or "embedding vector"; the decoder is used to convert the output of the encoder back to the original data, which can be realized by an inverse convolutional neural network, a recurrent neural network or other neural networks. The main function of the decoder is to restore the input data according to the encoding vector, and the output of the decoder is also called "reconstruction vector"; the discriminator can be a binary classifier, which is used to judge whether the generated spectrum feature is real or not. During pre-training, the generated spectrum and the real spectrum need to be input into the discriminator for discriminant training. During discriminant training, loss calculation needs to be performed based on the training source audio, the training target audio and the predicted audio to obtain the prediction deviation data, and then the model parameters of the audio processing model are updated based on the prediction deviation data.

[0117] Figure 6 According to some embodiments of the present application, the training content parameters include training prosody parameters and training spectrum parameters, and step S406 can include, but is not limited to, steps S601 to S602.

[0118] Step S601 performs language feature analysis on the training spectrum parameters to obtain a training hidden vector, and the training hidden vector is used to represent the sentence content of the training source audio;

[0119] Step S602, the emotion replacement vector, the training prosody parameter and the training hidden vector are integrated and spliced to obtain the predicted audio.

[0120] In steps S601 to S602 of some embodiments, the training spectral parameter is first analyzed for language features to obtain a training hidden vector, which is used to represent the sentence content of the training source audio, and then the emotion replacement vector, the training prosody parameter and the training hidden vector are integrated and spliced to obtain the predicted audio. It should be noted that,

[0121] The training prosody parameter is used to represent the prosody information and structural information of the song, such as the fundamental frequency F0; the training spectral parameter is used to represent the timbre and content of the speech of the song, such as the spectral parameter SP. It should be pointed out that, since the training spectral parameter reflects the timbre and content of the speech of the song in the training source audio, and the timbre part is to be excluded, therefore, the training hidden vector is obtained by extracting the part used to represent the sentence content of the training source audio through steps S601 to S602, and then the training hidden vector, the training prosody parameter and the emotion replacement vector are integrated and spliced to finally obtain the predicted audio. The timbre of the human voice can be converted while the prosody information and structural information of the source audio are preserved, and the accuracy and smoothness of the audio processing model in the song feature conversion are improved.

[0122] Reference Figure 7 According to some embodiments of the present application, the predicted deviation data includes a loss function value, and step S407 can include, but is not limited to, steps S701 to S704.

[0123] Step S701, based on the training source audio and the training hidden vector, a first loss calculation is performed to obtain an encoding loss element corresponding to the music emotion recognition network;

[0124] Step S702, based on the training source audio, the training target audio and the predicted audio, a second loss calculation is performed to obtain a decoding loss element corresponding to the variational autoencoder network;

[0125] Step S703, based on the encoding loss element and the decoding loss element, a loss is constructed to obtain a predicted loss function;

[0126] Step S704, based on the predicted loss function, a loss calculation is performed on the training source audio, the training target audio and the predicted audio to obtain a loss function value.

[0127] In step S701 of some embodiments, based on the training source audio and the training hidden vector, a first loss calculation is performed to obtain an encoding loss element corresponding to the music emotion recognition network. In some more specific embodiments, the spectral feature corresponding to the training source audio is represented as X s The prior distribution of the training hidden vector can be represented as pθ (Z), the posterior estimate corresponding to the predicted audio can be represented as q φ (Z|X s ), further, based on the training source audio and the training latent vector, a first loss calculation is performed to obtain an encoding loss element corresponding to the music emotion recognition network, wherein the encoding loss element can be specifically represented as:

[0128] -D KL (q φ (Z|X s )||p θ (Z))

[0129] wherein, D KL refers to the relative entropy calculation, that is, the calculation of KL divergence, wherein φ refers to the model parameter of the encoder in the variational adversarial network, and θ refers to the model parameter of the decoder in the variational adversarial network.

[0130] In step S702 of some embodiments, a second loss calculation is performed based on the training source audio, the training target audio and the predicted audio to obtain a decoding loss element corresponding to the variational adversarial network. It should be noted that in some more specific embodiments, the sound replacement identifier corresponding to the training target audio can be represented as y, and the model likelihood of the variational adversarial network can be represented as p θ (X s |Z,Y t ), further, based on the training source audio, the training target audio and the predicted audio, a second loss calculation is performed to obtain a decoding loss element corresponding to the variational adversarial network, wherein the decoding loss element can be specifically represented as:

[0131]

[0132] wherein, represents the expected value in a statistical sense.

[0133] In step S703 of some embodiments, a loss construction is performed based on the encoding loss element and the decoding loss element to obtain a predicted loss function. It should be noted that the loss construction is performed based on the encoding loss element and the decoding loss element to obtain the predicted loss function, which can be specifically represented as:

[0134]

[0135] In step S704 of some embodiments, a loss calculation is performed on the training source audio, the training target audio and the predicted audio based on the predicted loss function to obtain a loss function value. It should be noted that the loss function value is used to reflect the loss of the variational encoder in the current iteration round. Since the music emotion recognition network has been pre-trained, the loss of the variational encoder in the current iteration round can also be regarded as the loss of the audio processing model in the current iteration round.

[0136] The loss of the audio processing model in the current iteration round in each iteration training can be calculated via steps S701 to S704, so as to facilitate updating the model parameters of the audio processing model based on the prediction bias data after each iteration training, thereby pre-training the audio processing model information, so that the audio processing model gradually has the conversion capability of converting the singing features of the source singer to the singing features of the target singer, and the precision of the singing feature conversion of the present application is improved.

[0137] Referring to Figure 8 According to some embodiments of the present application, before step S704, the singing feature conversion method of the present application can further include, but is not limited to, steps S801 to S802.

[0138] Step S801, performing third loss calculation based on the training source audio, the training target audio and the prediction audio to obtain a discriminator loss element corresponding to the variational autoencoder;

[0139] Step S802, integrating the discriminator loss element into the prediction loss function based on a preset weight factor.

[0140] In steps S801 to S802 of some embodiments, first, the third loss calculation is performed based on the training source audio, the training target audio and the prediction audio to obtain a discriminator loss element corresponding to the variational autoencoder, and then the discriminator loss element is integrated into the prediction loss function based on a preset weight factor. It should be noted that the spectrum feature corresponding to the training target audio is represented as X t , the third loss calculation is performed based on the training source audio, the training target audio and the prediction audio to obtain a discriminator loss element corresponding to the variational autoencoder, wherein the discriminator loss element can be represented as:

[0141]

[0142] Wherein, the discriminator expression can be represented as D ψ (.), and the generator expression can be represented as G θ (.).

[0143] Further, the discriminator loss element is integrated into the prediction loss function based on a preset weight factor a, which can be represented as:

[0144]

[0145] Through the embodiments shown in steps S801 to S802, by integrating the discriminator loss element into the prediction loss function, the training efficiency of the discriminator in the variational autoencoder can be further improved.

[0146] Step S304, when the audio processing model meets the first preset condition in the iterative training, a pre-trained audio processing model is obtained. It should be noted that when the audio processing model meets the first preset condition in the iterative training, the pre-trained audio processing model can be obtained. It should be pointed out that the audio processing model meets the first preset condition in the iterative training, which means that the audio processing model in the current iteration round has reached the expectation of the pre-training process, and the accuracy of the singing voice feature conversion has reached the degree that can be applied to the actual scene. At this time, the audio processing model in the current iteration round is output, and the pre-trained audio processing model is obtained.

[0147] Figure 9 The electronic device 900 provided by the embodiments of the present application is shown. The electronic device 900 includes a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901, and the computer program is used to execute the singing voice feature conversion method described above when running.

[0148] The processor 901 and the memory 902 can be connected by a bus or other means.

[0149] The memory 902, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs, such as the singing voice feature conversion method described in the embodiments of the present application. The processor 901 runs the non-transitory software programs and instructions stored in the memory 902, thereby realizing the singing voice feature conversion method described above.

[0150] The memory 902 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function. The data storage area can store the singing voice feature conversion method described above. In addition, the memory 902 can include a high-speed random access memory 902, and can also include a non-transitory memory 902, such as at least one storage device, a flash memory device or other non-transitory solid state storage device. In some embodiments, the memory 902 can optionally include a memory 902 remotely arranged with respect to the processor 901, and these remote memories 902 can be connected to the electronic device 900 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0151] The non-transitory software programs and instructions required to realize the singing voice feature conversion method described above are stored in the memory 902, and when executed by one or more processors 901, the singing voice feature conversion method described above is executed, for example, the method steps S101 to S104 in Figure 1 , the method steps S201 to S205 in Figure 2 , the method steps S301 to S304 in Figure 3the method steps S301-S305 in the method 300, Figure 4 the method steps S401-S403 in the method 400, Figure 5 the method steps S501-S503 in the method 500, Figure 6 the method steps S601-S603 in the method 600, Figure 7 the method steps S701-S703 in the method 700, Figure 8 the method steps S801-S802 in the method 800.

[0152] The embodiments of the present application further provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used for executing the singing feature conversion method.

[0153] In an embodiment, the computer readable storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more control processors, for example, to execute the method steps S101-S104 in the method 100, Figure 1 the method steps S201-S205 in the method 200, Figure 2 the method steps S301-S305 in the method 300, Figure 3 the method steps S401-S403 in the method 400, Figure 4 the method steps S501-S503 in the method 500, Figure 5 the method steps S601-S603 in the method 600, Figure 6 the method steps S701-S703 in the method 700, Figure 7 the method steps S801-S802 in the method 800. Figure 8 The apparatus embodiments described above are merely illustrative, wherein the units described as separated components can or can not be physically separated, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0154]

[0155] ​Those skilled in the art can understand that all or some of the steps in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, storage devices storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and can include any information delivery media. It should also be understood that the various embodiments provided by the present application can be combined in any manner to achieve different technical effects.

[0156] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

Claims

1. A singing voice feature conversion method, comprising: obtaining source audio and target voice identification, the source audio comprising an audio track of a source singer; inputting the source audio and the target voice identification into a pre-trained audio processing model, the audio processing model comprising a vocoder, a music emotion recognition network, and a variational adversarial network; extracting acoustic features of the source audio based on the vocoder to obtain content feature parameters; extracting emotion information of the source audio based on the music emotion recognition network to obtain a source emotion feature vector; replacing the source emotion feature vector with a target emotion feature vector based on the target voice identification through the variational adversarial network; reconstructing the target emotion feature vector and the content feature parameters based on the variational adversarial network to obtain target audio after singing voice feature conversion.

2. The method of claim 1, before inputting the source audio and the target voice identification into the pre-trained audio processing model, the method further comprises pre-training the audio processing model, specifically comprising: obtaining a training data set comprising a first number of training audios, each of the training audios being configured with a corresponding training voice identification; grouping the first number of training audios to obtain a second number of training audio groups, each of the training audio groups comprising training audios of different singers singing the same audio track; inputting the second number of training audio groups into the original audio processing model, and iteratively training the audio processing model based on the training audios and the training voice identification; when the audio processing model meets a first preset condition in the iterative training, obtaining the pre-trained audio processing model.

3. The method of claim 2, inputting the second number of training audio groups into the original audio processing model and iteratively training the audio processing model based on the training audios and the training voice identification, comprising: in each round of iterative training, selecting an iteration audio group from the second number of training audio groups, and selecting a training source audio and a training target audio from the iteration audio group; extracting acoustic features of the training source audio based on the vocoder to obtain training content parameters; extracting emotion information of the training source audio based on the pre-trained music emotion recognition network to obtain an emotion initial vector; extracting emotion information of the training target audio based on the pre-trained music emotion recognition network to obtain an emotion replacement vector; determining the training voice identification configured for the training target audio as a voice replacement identification, and replacing the emotion initial vector with the emotion replacement vector based on the voice replacement identification through the variational adversarial network; reconstructing the emotion replacement vector and the training content parameters based on the variational adversarial network to obtain a predicted audio; calculating a loss based on the training source audio, the training target audio, and the predicted audio to obtain prediction deviation data; After each iteration, model parameters of the audio processing model are updated based on the predicted bias data.

4. The method of claim 3, wherein the training content parameters comprise training prosody parameters and training spectral parameters, and wherein reconstructing the emotion replacement vector and the training content parameters based on the variational autoencoder to obtain predicted audio comprises: performing linguistic feature analysis on the training spectral parameters to obtain training latent vectors, the training latent vectors being used to represent sentence content of the training source audio; and integrating and concatenating the emotion replacement vector, the training prosody parameters, and the training latent vectors to obtain the predicted audio.

5. The method of claim 4, wherein the predicted bias data comprises a loss function value, and wherein calculating a loss based on the training source audio, the training target audio, and the predicted audio to obtain predicted bias data comprises: performing first loss calculation based on the training source audio and the training latent vectors to obtain an encoding loss element corresponding to the music emotion recognition network; performing second loss calculation based on the training source audio, the training target audio, and the predicted audio to obtain a decoding loss element corresponding to the variational autoencoder; performing loss construction based on the encoding loss element and the decoding loss element to obtain a predicted loss function; and performing loss calculation based on the predicted loss function to obtain the training source audio, the training target audio, and the predicted audio to obtain the loss function value.

6. The method of claim 5, wherein before the performing loss calculation based on the predicted loss function to obtain the training source audio, the training target audio, and the predicted audio to obtain the loss function value, the method further comprises: performing third loss calculation based on the training source audio, the training target audio, and the predicted audio to obtain a discriminative loss element corresponding to the variational autoencoder; and integrating the discriminative loss element into the predicted loss function based on a preset weight factor.

7. The method of any one of claims 1 to 6, wherein the extracting content feature parameters based on the vocoder from the source audio comprises: extracting source prosody feature parameters based on the vocoder from the source audio, the source prosody feature parameters being used to represent prosody content of the source audio; extracting spectral feature parameters based on the vocoder from the source audio; and integrating the source prosody feature parameters and the spectral feature parameters to obtain the content feature parameters.

8. The method of claim 7, wherein the reconstructing the target emotion feature vector and the content feature parameters based on the variational autoencoder to obtain target audio after singing feature conversion comprises: obtaining the spectral feature parameters and the source prosody feature parameters from the content feature parameters; performing linguistic feature analysis on the spectral feature parameters to obtain sentence latent vectors, the sentence latent vectors being used to represent sentence content of the source audio; and integrating and concatenating the target emotion feature vector, the spectral feature parameters, and the sentence latent vectors to obtain the target audio after singing feature conversion. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ Integrate and splice the target emotional feature vector, the source prosody feature parameter and the sentence hidden vector to obtain the target audio.

9. An electronic device comprising: The memory stores a computer program, and the processor executes the computer program to realize the song feature conversion method in any one of claims 1 to 8. 10.A computer readable storage medium storing a computer program, and the computer program is executed by a processor to realize the song feature conversion method in any one of claims 1 to 8.

Images