Song generation method and device, electronic equipment and storage medium

Abstract

The invention provides a song generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a voice audio input by a target user and a unique identification number of a target song, carrying out the Mel spectrum feature extraction of the voice audio, and obtaining the real Mel spectrum feature of the target user, obtaining a song template corresponding to the unique identification number according to the unique identification number of the target song, inputting the real Mel spectrum feature of the target user and the song template into a preset song generation model to obtain a target Mel spectrum feature output by the song generation model, and generating the target song according to the target Mel spectrum feature, according to the embodiment of the invention, the real Mel spectrum feature of the target user and the song template corresponding to the target song can be effectively combined in the song generation process, so that the degree of dependence on the data volume of the user voice data is effectively reduced, and the song generation effect is effectively improved while the song generation convenience is improved.

Description

Song generation method, device, electronic device and storage medium Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a song generation method, device, electronic device, and storage medium. Background Art

[0002] Song synthesis involves generating singing audio based on lyrics and musical notation. The corresponding song synthesis algorithms have evolved from initial unit-based synthesis techniques to statistical parameter synthesis techniques and now to deep learning-based synthesis techniques. Song synthesis technology allows machines to sing, further enhancing the fun of human-computer interaction and therefore possessing high commercial value.

[0003] In the related art, when performing song synthesis, high requirements are usually placed on the quantity and quality of the training corpus, which makes the song generation process more complicated and the song generation effect cannot be guaranteed.

[0004] Summary of the Invention

[0005] The disclosed embodiments propose a song generation method, device, electronic device, and storage medium, which can be applied to the field of data processing technology. In the song generation process, the real mel-spectrogram features of the target user and the song template corresponding to the target song can be effectively combined to effectively reduce the dependence on the amount of user voice data, thereby improving the convenience of song generation while effectively improving the song generation effect.

[0006] In a first aspect, an embodiment of the present disclosure provides a song generation method, comprising:

[0007] Obtain the target user's input voice audio and the unique identification number of the target song;

[0008] Performing mel-spectrogram feature extraction on the speech audio to obtain the true mel-spectrogram features of the target user;

[0009] Acquire a song template corresponding to the unique identification number according to the unique identification number of the target song;

[0010] Inputting the target user's real mel-spectrogram features and the song template into a preset song generation model to obtain target mel-spectrogram features output by the song generation model, wherein the song generation model is trained by machine learning using a training set, the training set being from a plurality of sampled users, the training set including a plurality of samples, each sample corresponding to at least one sample, each sample including: singing audio picked up when the sampled user sings a certain song and lyrics text corresponding to the singing audio;

[0011] A target song is generated according to the target mel-spectrogram features.

[0012] In a second aspect, the present disclosure provides a method for training a song generation model, comprising:

[0013] Obtaining a training set, the training set being from a plurality of sampled users, the training set comprising a plurality of samples, one sampled user corresponding to at least one sample, each sample comprising: singing audio picked up when the sampled user sings a song and lyrics text corresponding to the singing audio;

[0014] Obtaining a pre-built initial neural network model, wherein the initial neural network model includes initial weight parameters and a loss function;

[0015] Obtaining a first sample from the training set, and inputting the first sample into the initial neural network model to obtain a true mel-spectrogram feature and a predicted mel-spectrogram feature, wherein the true mel-spectrogram feature represents the mel-spectrogram feature of the singing audio in the first sample, and the predicted mel-spectrogram feature represents the mel-spectrogram feature predicted by the initial neural network model;

[0016] Calculating the error between the predicted Mel spectrum feature and the true Mel spectrum feature according to the loss function;

[0017] Adjusting initial weight parameters of the initial neural network model according to the error to obtain an updated neural network model;

[0018] Subsequent samples are obtained one by one from the training set, and the subsequent samples are repeatedly input into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model.

[0019] In a third aspect, an embodiment of the present disclosure provides a song generation device, comprising: a first acquisition module, configured to acquire a voice audio input by a target user and a unique identification number of a target song;

[0020] A first processing module is configured to extract Mel-spectrogram features from the speech audio to obtain the true Mel-spectrogram features of the target user;

[0021] A second acquisition module is used to acquire a song template corresponding to the unique identification number of the target song according to the unique identification number of the target song;

[0022] a second processing module, configured to input the target user's real mel-spectrogram features and the song template into a preset song generation model to obtain target mel-spectrogram features output by the song generation model, wherein the song generation model is trained by machine learning using a training set, the training set being derived from a plurality of sampled users, the training set including a plurality of samples, each sample corresponding to at least one sample, each sample including: singing audio of the sampled user singing a song and lyrics corresponding to the singing audio;

[0023] A generation module is used to generate a target song according to the target mel spectrum features.

[0024] In a fourth aspect, an embodiment of the present disclosure provides a training device for a song generation model, characterized by comprising:

[0025] A third acquisition module is configured to acquire a training set, wherein the training set is from a plurality of sampled users, the training set comprising a plurality of samples, each sampled user corresponding to at least one sample, and each sample comprising: a singing audio of the sampled user singing a song and lyrics corresponding to the singing audio;

[0026] A fourth acquisition module is used to acquire a pre-built initial neural network model, wherein the initial neural network model includes initial weight parameters and a loss function;

[0027] a fifth acquisition module, configured to acquire a first sample from the training set and input the first sample into the initial neural network model to obtain a true mel-spectrogram feature and a predicted mel-spectrogram feature, wherein the true mel-spectrogram feature represents the mel-spectrogram feature of the singing audio in the first sample, and the predicted mel-spectrogram feature represents the mel-spectrogram feature predicted by the initial neural network model;

[0028] A third processing module is configured to calculate the error between the predicted mel spectrum feature and the true mel spectrum feature according to the loss function;

[0029] a fourth processing module, configured to adjust initial weight parameters of the initial neural network model according to the error to obtain an updated neural network model;

[0030] The sixth acquisition module is used to obtain subsequent samples one by one from the training set, and repeatedly input the subsequent samples into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model.

[0031] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the song generation method proposed in the embodiment of the first aspect of the present disclosure is implemented, or the training method of the song generation model proposed in the embodiment of the second aspect of the present disclosure is implemented.

[0032] In a sixth aspect, an embodiment of the present disclosure provides a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the song generation method proposed in the embodiment of the first aspect of the present disclosure, or implements the song generation model training method proposed in the embodiment of the second aspect of the present disclosure.

[0033] In the seventh aspect, an embodiment of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the song generation method proposed in the embodiment of the first aspect of the present disclosure is executed, or the song generation model training method proposed in the embodiment of the second aspect of the present disclosure is executed.

[0034] In summary, the song generation method, device, electronic device, storage medium, computer program, and computer program product provided in the embodiments of the present disclosure can achieve the following technical effects:

[0035] By obtaining the voice audio input by the target user and the unique identification number of the target song, Mel-spectrogram features are extracted from the voice audio to obtain the real Mel-spectrogram features of the target user, and the song template corresponding to the unique identification number of the target song is obtained according to the unique identification number of the target song. The real Mel-spectrogram features of the target user and the song template are input into a preset song generation model to obtain the target Mel-spectrogram features output by the song generation model. The target song is generated according to the target Mel-spectrogram features. The real Mel-spectrogram features of the target user and the song template corresponding to the target song can be effectively combined in the song generation process to effectively reduce the dependence on the data volume of the user's voice data, thereby effectively improving the song generation effect while improving the convenience of song generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the background technology, the drawings required for use in the embodiments of the present disclosure or the background technology will be described below.

[0037] FIG1 is a flow chart of a song generation method according to an embodiment of the present disclosure;

[0038] FIG2 is a flow chart of a song generation method according to another embodiment of the present disclosure;

[0039] FIG3 is a schematic diagram of a song template generation process proposed in an embodiment of the present disclosure;

[0040] FIG4 is a flow chart of a song generation method according to another embodiment of the present disclosure;

[0041] FIG5 is a schematic diagram of a timbre coding sub-model structure proposed in an embodiment of the present disclosure;

[0042] FIG6 is a schematic diagram of a song generation process according to an embodiment of the present disclosure;

[0043] FIG7 is a flow chart of a method for training a song generation model according to an embodiment of the present disclosure;

[0044] FIG8 is a training flow chart of an initial neural network model proposed in an embodiment of the present disclosure;

[0045] FIG9 is a schematic structural diagram of a song generation device proposed in an embodiment of the present disclosure;

[0046] FIG10 is a schematic structural diagram of a song generation device proposed in another embodiment of the present disclosure;

[0047] FIG11 is a schematic structural diagram of a training device for a song generation model according to an embodiment of the present disclosure;

[0048] FIG12 is a schematic structural diagram of a training device for a song generation model according to another embodiment of the present disclosure;

[0049] FIG13 illustrates a block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0051] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present disclosure. The singular forms "a," "an," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] To facilitate understanding, the terms involved in this disclosure are first introduced.

[0054] 1. MelPu

[0055] Mel-spectrograms are a common feature used in deep learning of speech. While regular spectrograms are linear, Mel-spectrograms, based on the characteristics of human hearing (which is more sensitive to low-frequency sounds and less able to distinguish high-frequency sounds), convert the frequency of regular spectrograms from linear to Mel-scale. The Mel-scale is a logarithmic scale, and humans are more sensitive to frequency on the Mel-scale.

[0056] 2. Phonemes

[0057] Phonemes are the smallest units of speech, defined by the natural properties of speech. They can be analyzed based on the pronunciation movements within a syllable, with each movement constituting a phoneme. Phonemes are divided into two main categories: vowels and consonants.

[0058] 3. Tone

[0059] Tone refers to the fact that different sounds always have unique characteristics in terms of waveform, and different objects have different vibration characteristics.

[0060] FIG1 is a flow chart of a song generation method according to an embodiment of the present disclosure.

[0061] It should be noted that the executor of the song generation method of this embodiment is a song generation device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, and the electronic device can include but is not limited to a terminal, a server, etc. For example, the terminal can be a smart phone, a smart TV, a smart watch, a smart car, etc.

[0062] As shown in FIG1 , the song generation method may include but is not limited to the following steps:

[0063] Step S101: Acquire the voice audio input by the target user and the unique identification number of the target song.

[0064] The target user refers to the user who will use the song generation method. The voice audio refers to the audio data input by the target user. This voice audio can be the target user's audio data or the audio data of another user, without limitation. The target song refers to the song to be generated by the song generation method. The unique identification number refers to the identification information corresponding to the target song, such as a number or name.

[0065] It is understandable that there may be multiple target songs. When the unique identification number of the target song is obtained, the target song can be accurately located during the song generation process.

[0066] In the embodiment of the present disclosure, when obtaining the voice audio input by the target user, an audio acquisition device may be pre-configured in the execution body of the embodiment of the present disclosure, and then the audio acquisition device obtains the voice audio of the target user. Alternatively, a data interface may be pre-configured in the execution body of the embodiment of the present disclosure, and a song generation request may be received via the data interface, and then the voice audio may be obtained by parsing the song generation request. There is no limitation to this.

[0067] In the embodiment of the present disclosure, when obtaining the unique identification number of the target song, a relationship table can be used, and the unique identification number corresponding to the target song can be recorded in the relationship table. Alternatively, a database can be established in advance based on the mapping relationship between multiple target songs and object unique identification numbers, and then the corresponding unique identification number can be obtained from the database based on the target song. There is no restriction on this.

[0068] Step S102: extracting mel-spectrogram features from the speech audio to obtain the true mel-spectrogram features of the target user.

[0069] Mel-spectrogram refers to the spectrum extracted from audio data, which is a logarithmic spectrum. Mel-spectrogram features refer to the characteristic information corresponding to the mel-spectrogram. It is understandable that the pitch of a sound perceived by the human ear is not linearly related to its actual frequency (Hz), making the use of mel-spectrogram features more consistent with the human ear's auditory characteristics. True mel-spectrogram features refer to the mel-spectrogram features obtained based on the aforementioned speech data.

[0070] In the embodiment of the present disclosure, when mel-spectrogram feature extraction is performed on speech audio to obtain the real mel-spectrogram feature of the target user, feature extraction of the speech audio can be achieved, thereby providing reliable reference data for the song generation process.

[0071] Step S103: According to the unique identification number of the target song, a song template corresponding to the unique identification number is obtained.

[0072] The song template refers to a template that describes the relevant information of the target song.

[0073] Optionally, in some embodiments, the song template may include lyrics text information and song melody information, the lyrics text information includes phoneme sequence and phoneme duration, and the song melody information includes song note sequence and song energy sequence. Thus, the representational content in the song template can be enriched to a great extent, thereby providing the song generation model with more comprehensive reference information of the target song, so as to effectively improve the applicability of the song template in the song generation process.

[0074] Lyrics text information refers to the textual information of the target song's lyrics. Song melody information can be used to describe the relevant information corresponding to the target song's melody. Phonemes are the smallest units of speech divided according to the natural properties of speech. A phoneme sequence is a sequence of multiple phonemes. Phoneme duration refers to the duration information corresponding to a phoneme. Notes are symbols used to record the progression of sounds of varying lengths. A song note sequence is a sequence of notes corresponding to the song's audio.

[0075] Among them, energy can refer to the energy contained in the song audio, such as sound intensity, and the song energy sequence can be used to describe the energy changes corresponding to the song audio at different time points.

[0076] In the embodiment of the present disclosure, when obtaining the song template corresponding to the unique identification number based on the unique identification number of the target song, multiple song templates can be obtained in advance, and then matching processing is performed based on the unique identification number and the multiple song templates to obtain the song template corresponding to the unique identification number. Alternatively, a third-party retrieval device can also obtain the song template corresponding to the unique identification number based on the unique identification number of the target song, and there is no restriction on this.

[0077] Step S104: Input the target user's real mel-spectrogram features and song template into a preset song generation model to obtain the target mel-spectrogram features output by the song generation model, wherein the song generation model is obtained through machine learning training using a training set, and the training set comes from multiple sampled users. The training set includes multiple samples, and one sampled user corresponds to at least one sample. Each sample includes: the singing audio picked up when the sampled user sings a certain song and the lyrics text corresponding to the singing audio.

[0078] The song generation model refers to the model used to process real mel-spectrogram features and song templates and output target mel-spectrogram features. This song generation model can be a neural network model. The target mel-spectrogram features refer to the mel-spectrogram features obtained by the song generation model by processing the real mel-spectrogram features of the target user and the song template. The training set refers to the sample set used during the training process of the song generation model.

[0079] A sampling user is a user who provides samples for the training process of the song generation model. Samples can refer to the singing audio and lyrics used for model training. Singing audio refers to the audio captured when a sampling user sings a song.

[0080] Optionally, in some embodiments, the song generation model includes: a timbre encoding sub-model, a text encoding sub-model and an acoustic decoding sub-model. The song generation model is obtained by jointly training the timbre encoding sub-model, the text encoding sub-model and the acoustic decoding sub-model using the same training set. Therefore, the structural rationality of the song generation model can be effectively improved. When the timbre encoding sub-model, the text encoding sub-model and the acoustic decoding sub-model are jointly trained using the same training set, the consistency between the sub-models can be effectively improved, thereby effectively improving the output accuracy of the obtained song generation model.

[0081] Timbre refers to the different characteristics of different sounds expressed in their waveforms. Just as different objects have different vibrational characteristics, different users have different timbres. The timbre encoding submodel processes the true mel-spectrogram features to generate the target user's timbre feature vector.

[0082] Among them, the text encoding sub-model refers to the model used to process the phoneme sequence to obtain the text feature vector corresponding to the target song.

[0083] The acoustic decoding sub-model refers to a model used to process multiple feature information to obtain target mel-spectrogram features. The acoustic decoding sub-model can be a decoder of a fast end-to-end and non-autoregressive synthesis system.

[0084] In the disclosed embodiment, when the target user's real mel-spectrogram features and song template are input into a preset song generation model to obtain the target mel-spectrogram features output by the song generation model, the relevant information of the real mel-spectrogram features and the song template can be quickly and accurately fused based on the song generation model, thereby effectively improving the model generation efficiency.

[0085] Step S105: Generate a target song based on the target mel-spectrogram features.

[0086] When generating a target song based on the target Mel-spectrogram features, the embodiment of the present disclosure may input the target Mel-spectrogram features into a vocoder, and the vocoder may parse and process the target spectrum features to obtain the target song.

[0087] For example, when generating a target song based on the target Mel spectrum feature, the target Mel spectrum feature can be input into a preset vocoder model to obtain a target linear spectrum feature; the linear spectrum feature is inverse Fourier transformed to obtain the audio data of the target song.

[0088] The vocoder model is a neural network model, trained through machine learning using a different training set than the song generation model. The vocoder model can be based on a Generative Adversarial Network (GAN), a GAN without distillation, or other techniques, and the training set can also be a commonly used training set in the prior art.

[0089] The training process of the vocoder model can be to input the real Mel spectrum features in a sample into the constructed initial model to obtain the predicted linear spectrum features, calculate the error between the predicted linear spectrum features and the real linear spectrum features in the sample through the loss function, modify the initial model weights according to the error, and input samples repeatedly until the loss function converges to obtain the trained vocoder model.

[0090] In the embodiment of the present disclosure, by obtaining the voice audio input by the target user and the unique identification number of the target song, mel-spectrogram features are extracted from the voice audio to obtain the real mel-spectrogram features of the target user, and a song template corresponding to the unique identification number of the target song is obtained according to the unique identification number of the target song. The real mel-spectrogram features of the target user and the song template are input into a preset song generation model to obtain the target mel-spectrogram features output by the song generation model, and the target song is generated according to the target mel-spectrogram features. The real mel-spectrogram features of the target user and the song template corresponding to the target song can be effectively combined in the song generation process to effectively reduce the dependence on the data volume of the user's voice data, thereby effectively improving the song generation effect while improving the convenience of song generation.

[0091] FIG2 is a flow chart of a song generation method proposed in another embodiment of the present disclosure.

[0092] As shown in FIG2 , the song generation method may include but is not limited to the following steps:

[0093] Step S201: Acquire the voice audio input by the target user and the unique identification number of the target song.

[0094] Step S202: extracting mel-spectrogram features from the speech audio to obtain the true mel-spectrogram features of the target user.

[0095] Step S203: Acquire a song template corresponding to the unique identification number according to the unique identification number of the target song.

[0096] The description of steps S201 to S203 can be found in the above embodiment and will not be repeated here.

[0097] Step S204: inputting the target user's real mel-spectrogram features into the timbre coding sub-model to obtain the target user's timbre feature vector.

[0098] The timbre feature vector refers to a vector used to represent the timbre feature corresponding to the target user.

[0099] Step S205: Input the phoneme sequence into the text encoding sub-model to obtain the text feature vector of the lyrics text in the song template.

[0100] The lyrics text refers to the text data in the song template that describes the corresponding lyrics information of the target song.

[0101] The text feature vector refers to a vector used to represent the text features corresponding to the lyrics text.

[0102] Optionally, in some embodiments, the song template is obtained by configuring the phoneme sequence, phoneme duration, song note sequence, song energy sequence, and unique identification number of the target song, wherein the phoneme sequence and phoneme duration of the target song are determined by the song audio and song lyrics of the target song, and the song note sequence and song energy sequence of the target song are determined by the song audio. Thus, the target song can be quickly located based on the unique identification number, so as to effectively improve the practicality of the obtained song template, and at the same time, effectively improve the accuracy of the song template's representation of information related to the target lyrics.

[0103] The song audio refers to the singing audio of the target song, and the song lyrics refers to the lyrics information of the target song.

[0104] Optionally, in some embodiments, the phoneme sequence includes: multiple phonemes obtained by parsing the song lyrics, and the phoneme duration includes: the first number of frames occupied by each phoneme in the song audio. This can effectively improve the adaptability between the obtained phoneme sequence and the song lyrics, and at the same time effectively improve the accuracy of the obtained first number of frames for the corresponding phonemes.

[0105] The first frame number refers to the number of video frames corresponding to the phonemes in the song audio.

[0106] Optionally, in some embodiments, the song energy sequence is obtained by quantizing the song energy features of the song audio, and the song note sequence is obtained by quantizing the song fundamental frequency features of the song audio. Thus, the clarity of the representation of the song energy features and the song fundamental frequency features by the obtained song energy sequence and song note sequence can be effectively improved based on the quantization processing. At the same time, the song energy sequence and song note sequence obtained by the quantization processing can provide reliable reference data for subsequent calculation processes.

[0107] The song energy feature can be used to describe the relevant features corresponding to the song energy, and the song fundamental frequency feature can be used to describe the relevant features corresponding to the song fundamental frequency.

[0108] Optionally, in some embodiments, the song energy feature includes: multiple energy values, the song energy sequence is formed based on multiple range coding values, the range coding value is obtained by one-hot encoding the energy range corresponding to the energy value, and the song energy feature can be effectively expanded based on the one-hot encoding process to distinguish multiple energy values ​​based on the obtained multiple range coding values. When the song energy sequence is formed based on multiple range coding values, the representation effect of the obtained song energy sequence on the song energy feature can be effectively improved.

[0109] The energy value may refer to a numerical value corresponding to the energy of the song, and the energy range may refer to a value range corresponding to the energy value, such as 0-10.

[0110] Among them, one-hot encoding, which can also be called one-bit effective encoding, can use an N-bit state register to encode N states. Each state has its own independent register bit, and at any time, only one of them is effective.

[0111] The range encoding value refers to the encoding value obtained by one-hot encoding the energy range.

[0112] For example, encode six states: suppose the natural sequence codes of the six states are: 000, 001, 010, 011, 100, 101.

[0113] The corresponding configuration of one-hot encoding is: 000001, 000010, 000100, 001000, 010000, 100000.

[0114] Optionally, in some embodiments, the fundamental frequency characteristics of the song include: multiple fundamental frequency values, and the song note sequence includes a note symbol corresponding to each fundamental frequency value. Thus, the song note sequence can effectively combine the correspondence between the fundamental frequency value and the note symbol to adapt to personalized application scenarios, thereby effectively improving the applicability of the obtained song note sequence in the song generation process.

[0115] The fundamental frequency value refers to the numerical value corresponding to the fundamental frequency of the song. The note number refers to the number corresponding to the note, which can be obtained based on a relevant database in the music field.

[0116] For example, as shown in FIG3 , FIG3 is a schematic diagram of a song template generation process proposed in an embodiment of the present disclosure. The initial data of the song template may include the song audio and song lyrics corresponding to the target song. The song template generation process may include: (1) processing the song lyrics based on a text transcription method to obtain a phoneme sequence corresponding to the target song; (2) processing the obtained song phoneme sequence and song audio based on forced alignment to obtain the phoneme duration of the target song. Here, the processing may be performed by a forced alignment method, or manual calibration may be performed after the forced alignment operation to improve the accuracy of the obtained phoneme duration; (3) extracting the phoneme duration based on an acoustic feature extraction method. Process the song audio to obtain the song energy features and song fundamental frequency features corresponding to the target song, and then change the values ​​of the energy and pitch corresponding to the song based on the energy trajectory translation and the fundamental frequency trajectory translation to improve the flexibility of the song template; (4) Quantify the song energy features and the song fundamental frequency features to obtain the song energy sequence and the song note sequence; (5) Generate a song template based on the phoneme sequence, the phoneme duration, the song energy sequence and the song note sequence; (6) After the song template is generated, a unique identification number of the target song can be generated for the song template to facilitate retrieval of the song template based on the unique identification number during the song generation process.

[0117] Step S206: performing duration regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain a frame-level text feature vector and a frame-level timbre feature vector.

[0118] The frame-level text feature vector refers to a vector describing the text features corresponding to multiple audio frames, while the frame-level timbre feature vector refers to a vector describing the timbre features corresponding to multiple audio frames.

[0119] It is understandable that the same phoneme may include multiple audio frames, and the multiple audio frames corresponding to the same phoneme have high similarity. When the text feature vector and the timbre feature vector are time-regulated according to the phoneme duration to obtain the frame-level text feature vector and the frame-level timbre feature vector, the phoneme-level text feature vector and the timbre feature vector can be converted into the frame-level text feature vector and the timbre feature vector by copying, so as to facilitate the subsequent addition processing of the frame-level text feature vector and the frame-level timbre feature vector.

[0120] Step S207: The frame-level text feature vector, the frame-level timbre feature vector and the song melody information are added together and input into the acoustic decoding sub-model to obtain the target mel-spectrogram feature.

[0121] Here, addition refers to the addition operation of dimensions. Assuming that the dimensions of the frame-level text feature vector, frame-level timbre feature vector, song note sequence, and song energy sequence are all 10 dimensions, addition is the addition operation of the values ​​of the corresponding dimensions.

[0122] That is to say, after obtaining the song template corresponding to the unique identification number of the target song according to the unique identification number of the present disclosure, the real mel-spectrogram feature of the target user can be input into the timbre encoding sub-model to obtain the timbre feature vector of the target user, and the phoneme sequence can be input into the text encoding sub-model to obtain the text feature vector of the lyrics text in the song template. The text feature vector and the timbre feature vector are time-warped according to the phoneme duration to obtain the frame-level text feature vector and the frame-level timbre feature vector. The frame-level text feature vector, the frame-level timbre feature vector and the song melody information are added and input into the acoustic decoding sub-model to obtain the target mel-spectrogram feature. Thus, the feature extraction of the real mel-spectrogram feature and the phoneme sequence can be quickly realized based on the timbre encoding sub-model and the text encoding sub-model, and the corresponding timbre features and text features can be quantized in the form of vectors. Then, the text feature vector and the timbre feature vector are time-warped based on the phoneme duration, which can effectively improve the consistency between the obtained frame-level text feature vector and the frame-level timbre feature vector, so as to effectively improve the processing effect of the acoustic decoding sub-model on the frame-level text feature vector and the frame-level timbre feature vector.

[0123] Step S208: Generate a target song based on the target mel-spectrogram features.

[0124] The description of step S208 can be found in the above embodiment and will not be repeated here.

[0125] In this embodiment, the target user's real Mel-spectrogram features are input into the timbre encoding sub-model to obtain the target user's timbre feature vector, the phoneme sequence is input into the text encoding sub-model to obtain the text feature vector of the lyrics text in the song template, the text feature vector and the timbre feature vector are time-warped according to the phoneme duration to obtain the frame-level text feature vector and the frame-level timbre feature vector, the frame-level text feature vector, the frame-level timbre feature vector and the song melody information are added and input into the acoustic decoding sub-model to obtain the target Mel-spectrogram features. Thus, the feature extraction of the real Mel-spectrogram features and the phoneme sequence can be quickly realized based on the timbre encoding sub-model and the text encoding sub-model, and the corresponding timbre features and text features can be quantized in the form of vectors, and then the text feature vector and the timbre feature vector are time-warped based on the phoneme duration, which can effectively improve the consistency between the obtained frame-level text feature vector and the frame-level timbre feature vector, so as to effectively improve the processing effect of the acoustic decoding sub-model on the frame-level text feature vector and the frame-level timbre feature vector.

[0126] FIG4 is a flow chart of a song generation method proposed in another embodiment of the present disclosure.

[0127] As shown in FIG4 , the song generation method may include but is not limited to the following steps:

[0128] Step S401: Acquire the voice audio input by the target user and the unique identification number of the target song.

[0129] Step S402: extracting mel-spectrogram features from the speech audio to obtain the true mel-spectrogram features of the target user.

[0130] Step S403: According to the unique identification number of the target song, a song template corresponding to the unique identification number is obtained.

[0131] The description of steps S401 to S403 can be found in the above embodiment and will not be repeated here.

[0132] Step S404: Input the target user's real mel-spectrogram features into the reference encoder to obtain the target user's timbre latent space distribution vector.

[0133] The reference encoder may refer to an encoder used to process the real Mel spectrum feature to obtain the timbre latent space distribution vector, and the timbre latent space distribution vector output by the reference encoder may refer to the hidden layer variable corresponding to the real Mel spectrum feature.

[0134] It can be understood that the timbre latent space distribution vector obeys a spherical Gaussian distribution. In the embodiment of the present disclosure, the reference encoder can output the mean and variance corresponding to the spherical Gaussian distribution while outputting the timbre latent space distribution vector of the target user.

[0135] Step S405: Input the timbre latent space distribution vector into the autoregressive encoder to obtain the timbre distribution vector of the target user, wherein the timbre distribution vector is obtained by sampling the timbre latent space distribution vector by the autoregressive encoder.

[0136] Among them, the autoregressive encoder refers to an encoder used to process the timbre latent space distribution vector to obtain the timbre distribution vector.

[0137] It is understandable that the structures of the above-mentioned reference encoder and autoregressive encoder can be multiple layers of linear layers or convolutional layers, and there is no limitation on this.

[0138] Step S406: Using the timbre distribution vector as the timbre feature vector of the target user.

[0139] That is to say, the timbre encoding sub-model may include: a reference encoder and an autoregressive encoder. After obtaining the song template corresponding to the unique identification number of the target song, the real Mel-spectrogram features of the target user may be input into the reference encoder to obtain the timbre latent space distribution vector of the target user. The timbre latent space distribution vector is input into the autoregressive encoder to obtain the timbre distribution vector of the target user, wherein the timbre distribution vector is obtained by sampling the timbre latent space distribution vector by the autoregressive encoder. The timbre distribution vector is used as the timbre feature vector of the target user. In this way, the redundant information in the obtained timbre feature vector can be effectively reduced, and the more complex real Mel-spectrogram features can be converted into vector form, thereby effectively improving the practicality of the obtained timbre feature vector.

[0140] For example, as shown in FIG5 , FIG5 is a schematic diagram of a timbre coding sub-model structure proposed in an embodiment of the present disclosure, wherein the random sampling point ε refers to a random sampling point of a Gaussian distribution, which can be expressed as ε~N(0,I).

[0141] After receiving the real Mel spectrum features, the timbre coding sub-model can be processed by a reference encoder to obtain the timbre latent space distribution vector h and two parameters. The two parameters can be used as the mean a1 and variance b1 of the Gaussian distribution respectively. The above-mentioned random sampling point ε is combined with the mean a1 and variance b1 for processing to obtain a random sampling point z (z = b1⊙ε+a1, where ⊙ refers to matrix multiplication) that approximates the post-experimental distribution. The random sampling point z and the timbre latent space distribution vector h can be processed by an autoregressive encoder to obtain the mean a2 and variance b2 corresponding to the random sampling point z. Then, based on the random sampling point z, the mean a2 and the variance b2, the timbre feature vector s (s = b2⊙z+a2) can be obtained.

[0142] It is understandable that the timbre encoding sub-model can be a sampling process based on the Inverse Autoregressive Flow (IAF), which belongs to the standardized flow. The standardized flow can generate a distribution that is easy to sample. Through a series of reversible transformation operations, the standardized flow can convert a complex input distribution into a probability distribution that is easy to handle. The output distribution is usually selected as an isotropic unit Gaussian distribution, that is, a spherical unit Gaussian distribution, which allows smooth interpolation and effective sampling. The timbre feature vector is learned by using the inverse autoregressive flow method. The generated timbre latent space distribution vector h can obey the spherical Gaussian distribution, so that the timbre feature vector can be sampled on this distribution. It can also be used for unprocessed users to learn and obtain a more accurate vector distribution. In both the training and inference stages, sampling is performed from the spherical Gaussian distribution to represent the timbre feature vector, thereby ensuring the consistency of the training and inference stages and being more suitable for users outside the training set. At the same time, sampling the user's corresponding timbre latent space distribution vector h instead of averaging further increases the convergence of the user space, thereby allowing smoother interpolation between timbre feature vectors. In other words, the timbre feature vector of a user outside the training set can be learned from a sentence of audio from the user outside the training set.

[0143] Step S407: Input the phoneme sequence into the text encoding sub-model to obtain the text feature vector of the lyrics text in the song template.

[0144] The description of step S407 can be found in the above embodiment and will not be repeated here.

[0145] Step S408: Determine the initial text code corresponding to each phoneme in the phoneme sequence from the text feature vector.

[0146] The initial text encoding refers to the text encoding contained in the text feature vector.

[0147] In the embodiment of the present disclosure, when the initial text encoding corresponding to each phoneme in the phoneme sequence is determined from the text feature vector, reliable reference data can be provided for the subsequent determination of the target text encoding.

[0148] Step S409: Determine the first frame number corresponding to the phoneme according to the phoneme duration.

[0149] The first frame number refers to the number of video frames corresponding to each phoneme. For example, the phoneme duration corresponding to a phoneme may be 25ms. If an audio frame is set to 5ms, then one phoneme includes information of 5 frames.

[0150] Step S410: copying the initial text code, and performing splicing processing on the copied initial text codes of the first number of frames to obtain the target text code.

[0151] The target text code refers to the text code obtained by splicing the initial text codes of the first number of frames.

[0152] It is understandable that the phoneme duration corresponding to a phoneme may be small, so there may be more redundant information between multiple audio frames corresponding to the same phoneme. When the initial text code is copied and the initial text code of the first number of frames obtained by copying is spliced ​​to obtain the target text code, the practicality of the obtained target text code can be effectively improved.

[0153] Step S411: Generate a frame-level text feature vector based on multiple target text codes.

[0154] That is to say, after the embodiment of the present disclosure inputs the phoneme sequence into the text encoding sub-model and obtains the text feature vector of the lyrics text in the song template, it can determine the initial text encoding corresponding to each phoneme in the phoneme sequence from the text feature vector, determine the first frame number corresponding to the phoneme according to the phoneme duration, copy the initial text encoding, and splice the initial text encoding of the copied first frame number to obtain the target text encoding, and form a frame-level text feature vector based on multiple target text encodings. Since the time range corresponding to each phoneme is small and the representation content of different audio frames in the same phoneme has a high similarity, when the initial text encoding is copied and the initial text encoding of the copied first frame number is spliced ​​to obtain the target text encoding, the computational cost can be greatly reduced, thereby effectively improving the efficiency of determining the frame-level text feature vector.

[0155] Step S412: Determine the second frame number of the speech audio according to the phoneme duration.

[0156] The second number of frames refers to the number of frames of the speech audio determined based on the phoneme duration.

[0157] Step S413: copying the timbre feature vector, and performing concatenation processing on the copied timbre feature vectors of the second number of frames to obtain a frame-level timbre feature vector.

[0158] That is to say, after encoding multiple target texts and forming a frame-level text feature vector, the embodiment of the present disclosure can determine the second frame number of speech audio according to the phoneme duration, copy the timbre feature vector, and splice the timbre feature vectors of the copied second frame number to obtain a frame-level timbre feature vector. Thus, the obtained frame-level timbre feature vector can effectively represent the relevant feature information of the speech to be processed from the level of audio frames, so as to effectively improve the adaptability between the obtained frame-level timbre feature vector and the frame-level text feature vector, and at the same time effectively improve the representation effect of the obtained frame-level timbre feature vector.

[0159] Step S414: The frame-level text feature vector, the frame-level timbre feature vector, and the song melody information are added together and input into the acoustic decoding sub-model to obtain the target mel-spectrogram feature.

[0160] Step S415: Generate a target song based on the target mel-spectrogram features.

[0161] The description of step S414 to step S415 can be found in the above embodiment and will not be repeated here.

[0162] For example, as shown in FIG6 , FIG6 is a schematic diagram of a song generation process proposed in an embodiment of the present disclosure. When a new user provides a speech audio, the corresponding operation process of the song generation model may include: (1) processing the speech audio based on the acoustic feature extraction method to obtain a true mel-spectrogram feature; (2) inputting the true mel-spectrogram feature into the timbre encoding sub-model to obtain a timbre feature vector; (3) inputting the phoneme sequence in the song template into the text encoding sub-model to obtain a text feature vector of the target song; (4) inputting the phoneme duration, timbre feature vector, and text feature vector into the duration regularization sub-module to obtain a frame-level text feature vector and a frame-level timbre feature vector; (5) adding the frame-level text feature vector, the frame-level timbre feature vector, the song note sequence, and the song energy sequence and inputting them into the acoustic decoding sub-model to obtain a target mel-spectrogram feature; (6) inputting the obtained target spectrum feature into the vocoder to obtain the target song. The vocoder may be a neural network vocoder.

[0163] That is to say, in the song generation process of the embodiment of the present disclosure, multiple users can share the pre-trained song generation model, and obtain the song performed by the user based on a piece of audio of the user, thereby effectively improving the convenience of the song generation process, effectively reducing computing resources, and lowering storage costs.

[0164] In this embodiment, the target user's real Mel spectrum features are input into a reference encoder to obtain the target user's timbre latent space distribution vector, and the timbre latent space distribution vector is input into an autoregressive encoder to obtain the target user's timbre distribution vector, wherein the timbre distribution vector is obtained by sampling the timbre latent space distribution vector by the autoregressive encoder. The timbre distribution vector is used as the target user's timbre feature vector, thereby effectively reducing redundant information in the obtained timbre feature vector and converting the more complex real Mel spectrum features into a vector form, thereby effectively improving the practicality of the obtained timbre feature vector. The initial text code corresponding to each phoneme in the phoneme sequence is determined from the text feature vector, the first frame number corresponding to the phoneme is determined according to the phoneme duration, the initial text code is copied, and the initial text code of the copied first frame number is spliced ​​to obtain the target text code. A frame-level text feature vector is formed based on multiple target text codes. Since the time range corresponding to each phoneme is small and the representation content of different audio frames in the same phoneme has a high similarity, when the initial text code is copied and the initial text code of the copied first frame number is spliced ​​to obtain the target text code, the calculation cost can be greatly reduced, thereby effectively improving the efficiency of determining the frame-level text feature vector. By determining the second frame number of speech audio according to the phoneme duration, copying the timbre feature vector, and splicing the timbre feature vector of the copied second frame number to obtain the frame-level timbre feature vector, the obtained frame-level timbre feature vector can effectively represent the relevant feature information of the speech to be processed from the level of audio frames, so as to effectively improve the adaptability between the obtained frame-level timbre feature vector and the frame-level text feature vector, and at the same time effectively improve the representation effect of the obtained frame-level timbre feature vector.

[0165] FIG7 is a flowchart of a method for training a song generation model according to an embodiment of the present disclosure.

[0166] It should be noted that the executor of the training method of the song generation model of this embodiment is the training device of the song generation model, which can be implemented by software and / or hardware. The device can be configured in an electronic device, and the electronic device can include but is not limited to a terminal, a server, etc. For example, the terminal can be a smart phone, a smart TV, a smart watch, a smart car, etc.

[0167] As shown in FIG7 , the training method of the song generation model may include but is not limited to the following steps:

[0168] Step S701: Obtain a training set. The training set comes from multiple sampling users. The training set includes multiple samples. One sampling user corresponds to at least one sample. Each sample includes: the singing audio picked up when the sampling user sings a song and the lyrics text corresponding to the singing audio.

[0169] In the embodiment of the present disclosure, when obtaining the training set, a communication link between the execution entity of the embodiment of the present disclosure and the big data server can be established in advance, and then the training set can be obtained from the big data server, or the training set can be obtained from multiple sampling users based on the sample collection device, and there is no limitation on this.

[0170] Step S702: Obtain a pre-built initial neural network model, which includes initial weight parameters and a loss function.

[0171] A neural network model is a complex network system formed by extensively interconnecting a large number of simple processing units (called neurons). It reflects many fundamental characteristics of human brain function. The initial neural network model refers to the neural network model to be trained. The initial weight parameters refer to the weight parameters to be iteratively updated during model training. The loss function can be used to describe the error between the predicted mel-spectrogram features output by the initial neural network model during training and the true mel-spectrogram features.

[0172] In the disclosed embodiment, the model performance can be evaluated in real time during the model training process based on the loss function to timely determine whether the model has converged.

[0173] Step S703: Obtain the first sample from the training set and input the first sample into the initial neural network model to obtain the true mel-spectrogram feature and the predicted mel-spectrogram feature. The true mel-spectrogram feature represents the mel-spectrogram feature of the singing audio in the first sample, and the predicted mel-spectrogram feature represents the mel-spectrogram feature predicted by the initial neural network model.

[0174] The first sample refers to the first sample among multiple samples in the training set that is used for model training.

[0175] When obtaining the first sample from the training set, the embodiment of the present disclosure may randomly obtain a sample from the training set as the first sample, or may obtain the first sample from the training set based on the numbering information of multiple samples in the training set, without limitation.

[0176] Optionally, in some embodiments, when the first sample is input into the initial neural network model to obtain the true Mel-spectrogram features and the predicted Mel-spectrogram features, the lyrics text in the first sample may be transcribed to obtain a phoneme sequence, and the singing audio pair in the first sample may be aligned according to the phoneme sequence to obtain the phoneme duration, and the singing audio in the first sample may be subjected to acoustic feature extraction to obtain the true Mel-spectrogram features, audio energy and fundamental frequency trajectory of the first sample, and the phoneme sequence may be input into the initial text encoding sub-model to obtain the text feature vector of the first sample, and the true Mel-spectrogram features of the first sample may be input into the initial timbre encoding sub-model to obtain the timbre feature vector of the first sample. , the text feature vector and the timbre feature vector are time-regularized according to the phoneme duration to obtain the frame-level text feature vector and the frame-level timbre feature vector, the frame-level text feature vector, the frame-level timbre feature vector, the audio energy, and the fundamental frequency trajectory are added and input into the initial acoustic decoding sub-model to obtain the predicted Mel spectrum feature of the first sample. Therefore, different methods can be used to extract multiple features from the lyrics text and singing audio during the model training process, and the obtained multiple features can be converted into the form of vectors. While quantizing the features, it is convenient to add the obtained multiple feature vectors to achieve feature fusion, which can effectively improve the accuracy of the description of the sample features by the obtained predicted Mel spectrum feature.

[0177] The initial text encoding sub-model refers to the text encoding sub-model to be trained. The initial timbre encoding sub-model refers to the timbre encoding sub-model to be trained. The initial acoustic decoding sub-model refers to the acoustic decoding sub-model to be trained.

[0178] Among them, audio energy refers to the energy information corresponding to the singing audio in the first sample.

[0179] The fundamental frequency trajectory refers to the trajectory information of the fundamental frequency corresponding to the singing audio in the first sample.

[0180] Step S704: Calculate the error between the predicted mel spectrum feature and the true mel spectrum feature according to the loss function.

[0181] Among them, the error can be used to describe the difference information between the predicted Mel spectrum features and the true Mel spectrum features.

[0182] In the embodiment of the present disclosure, when the error between the predicted mel spectrum feature and the true mel spectrum feature is calculated based on the loss function, the output accuracy of the initial neural network model can be evaluated in real time to determine the model performance. The obtained error can provide reliable reference data for determining the direction of model optimization.

[0183] Step S705: adjusting the initial weight parameters of the initial neural network model according to the error to obtain an updated neural network model.

[0184] In the embodiment of the present disclosure, when the initial weight parameters of the initial neural network model are adjusted according to the error, the initial weight parameters can be accurately adjusted based on the error, thereby effectively improving the training effect of the neural network model.

[0185] Step S706: Obtain subsequent samples one by one from the training set, and repeatedly input the subsequent samples into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model.

[0186] Subsequent samples refer to samples other than the first sample in the training set.

[0187] For example, as shown in FIG8 , FIG8 is a training flow chart of an initial neural network model proposed in an embodiment of the present disclosure. The initial neural network model may include an initial timbre encoding sub-model, an initial text encoding sub-model, and an initial acoustic decoding sub-model. The training process may include: (1) the lyrics of the song are transcribed through text to obtain a corresponding phoneme sequence, and the obtained phoneme sequence is processed through the initial text encoding sub-model to obtain a corresponding text feature vector; (2) the text feature vector and the song phoneme duration are processed based on forced alignment to obtain an initial text encoding; (3) based on acoustic feature extraction By processing song audio, real mel spectrum features, song energy features and song fundamental frequency features can be obtained; (4) the real mel spectrum features are processed through the initial timbre coding sub-model to obtain the timbre feature vector; (5) multiple energy values ​​in the song audio energy features are divided into different energy bands (for example, energy values ​​ranging from 0 to 10 can be divided into 10 or 20 energy bands based on the application environment), and the song energy features are processed based on the one-hot encoding method to obtain the song energy sequence; (6) based on relevant data in the music field, the song fundamental frequency features are quantized to obtain the song note sequence. For example, the note number corresponding to the fundamental frequency of 261.63 Hz is 60, and the note number corresponding to the fundamental frequency of 277.18 Hz is 61. (7) Using the time warping method, the initial encoded text is processed based on the song phoneme duration to obtain a frame-level text feature vector. (8) The time warping sub-model processes the timbre feature vector based on the song phoneme duration to obtain a frame-level timbre feature vector. (9) The above-mentioned frame-level text feature vector, frame-level timbre feature vector, song energy sequence, and song note sequence are input into the acoustic decoding sub-model to obtain the predicted mel-spectrogram feature. (10) The loss function of the song generation model is determined based on the true mel-spectrogram feature and the predicted spectral feature. Through this loss function, the various weight parameters in the song generation model can be iteratively updated by sampling gradient back propagation to make the loss function converge.

[0188] In an embodiment of the present disclosure, a training set is obtained, the training set being from multiple sampling users and including multiple samples, with each sampling user corresponding to at least one sample. Each sample includes: singing audio picked up by the sampling user when singing a certain song and lyrics text corresponding to the singing audio. A pre-built initial neural network model is obtained, the initial neural network model including initial weight parameters and a loss function. A first sample is obtained from the training set and input into the initial neural network model to obtain true mel-spectrogram features and predicted mel-spectrogram features. The true mel-spectrogram features represent the mel-spectrogram features of the singing audio in the first sample, and the predicted mel-spectrogram features represent the mel-spectrogram features predicted by the initial neural network model. The error between the predicted mel-spectrogram features and the true mel-spectrogram features is calculated based on the loss function. The initial weight parameters of the initial neural network model are adjusted based on the error to obtain an updated neural network model. Subsequent samples are obtained one by one from the training set and repeatedly input into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model. Thus, the error between the predicted mel-spectrogram features and the true mel-spectrogram features output by the model can be determined in real time based on the loss function during the model training process, thereby providing a reliable basis for judging model convergence and effectively improving the output accuracy of the song generation model.

[0189] FIG9 is a schematic structural diagram of a song generation device proposed in an embodiment of the present disclosure.

[0190] As shown in FIG9 , the song generating device 90 includes:

[0191] The first acquisition module 901 is used to obtain the voice audio input by the target user and the unique identification number of the target song;

[0192] The first processing module 902 is used to extract Mel-spectrogram features from the speech audio to obtain the real Mel-spectrogram features of the target user;

[0193] The second acquisition module 903 is used to acquire a song template corresponding to the unique identification number of the target song according to the unique identification number;

[0194] The second processing module 904 is configured to input the target user's actual mel-spectrogram features and the song template into a preset song generation model to obtain target mel-spectrogram features output by the song generation model, wherein the song generation model is trained through machine learning using a training set, the training set being derived from multiple sampled users. The training set includes multiple samples, with each sampled user corresponding to at least one sample, and each sample including: a singing audio of the sampled user singing a particular song and lyrics corresponding to the singing audio;

[0195] The generation module 905 is used to generate a target song according to the target mel-spectrogram features.

[0196] In some embodiments of the present disclosure, as shown in FIG10 , FIG10 is a structural diagram of a song generation device proposed in another embodiment of the present disclosure, and the song generation model includes: a timbre encoding sub-model, a text encoding sub-model, and an acoustic decoding sub-model; the song generation model is obtained by jointly training the timbre encoding sub-model, the text encoding sub-model, and the acoustic decoding sub-model using the same training set.

[0197] In some embodiments of the present disclosure, the song template includes lyrics text information and song melody information; the lyrics text information includes phoneme sequence and phoneme duration; and the song melody information includes song note sequence and song energy sequence.

[0198] In some embodiments of the present disclosure, the second processing module 904 includes: a first processing sub-module 9041, which is used to input the real mel-spectrogram features of the target user into the timbre encoding sub-model to obtain the timbre feature vector of the target user; a second processing sub-module 9042, which is used to input the phoneme sequence into the text encoding sub-model to obtain the text feature vector of the lyrics text in the song template; a third processing sub-module 9043, which is used to perform time length regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain a frame-level text feature vector and a frame-level timbre feature vector; a fourth processing sub-module 9044, which is used to add the frame-level text feature vector, the frame-level timbre feature vector and the song melody information and input them into the acoustic decoding sub-model to obtain the target mel-spectrogram features.

[0199] In some embodiments of the present disclosure, it is characterized in that the first processing sub-module 9041 is specifically used to: input the real Mel-spectrogram features of the target user into the reference encoder to obtain the timbre latent space distribution vector of the target user; input the timbre latent space distribution vector into the autoregressive encoder to obtain the timbre distribution vector of the target user, wherein the timbre distribution vector is obtained by sampling the timbre latent space distribution vector by the autoregressive encoder; and use the timbre distribution vector as the timbre feature vector of the target user.

[0200] In some embodiments of the present disclosure, the third processing submodule 9043 is specifically used to: determine the initial text code corresponding to each phoneme in the phoneme sequence from the text feature vector; determine the first frame number corresponding to the phoneme based on the phoneme duration; copy the initial text code, and splice the initial text code of the copied first frame number to obtain the target text code; and form a frame-level text feature vector based on multiple target text codes.

[0201] In some embodiments of the present disclosure, the third processing submodule 9043 is further configured to: determine a second frame number of speech audio based on the phoneme duration; copy the timbre feature vector, and concatenate the copied second frame number of timbre feature vectors to obtain a frame-level timbre feature vector. In some embodiments of the present disclosure, the song template is configured by the phoneme sequence, phoneme duration, song note sequence, song energy sequence, and a unique identification number of the target song, wherein the phoneme sequence and phoneme duration of the target song are determined by the song audio and song lyrics of the target song, and the song note sequence and song energy sequence of the target song are determined by the song audio.

[0202] In some embodiments of the present disclosure, the phoneme sequence includes: a plurality of phonemes obtained by parsing the lyrics of a song, and the phoneme duration includes: the first frame number occupied by each phoneme in the song audio.

[0203] In some embodiments of the present disclosure, the song energy sequence is obtained by quantizing the song energy features of the song audio, and the song note sequence is obtained by quantizing the song fundamental frequency features of the song audio.

[0204] In some embodiments of the present disclosure, the song energy feature includes: multiple energy values; the song energy sequence is formed based on multiple range coding values, and the range coding value is obtained by performing one-hot encoding processing on the energy range corresponding to the energy value.

[0205] In some embodiments of the present disclosure, the song fundamental frequency feature includes: a plurality of fundamental frequency values; and the song note sequence includes a note symbol corresponding to each fundamental frequency value.

[0206] It should be noted that the above explanation of the song generation method is also applicable to the song generation device of this embodiment and will not be repeated here.

[0207] In this embodiment, by obtaining the voice audio input by the target user and the unique identification number of the target song, mel-spectrogram features are extracted from the voice audio to obtain the real mel-spectrogram features of the target user, and the song template corresponding to the unique identification number of the target song is obtained according to the unique identification number of the target song. The real mel-spectrogram features of the target user and the song template are input into a preset song generation model to obtain the target mel-spectrogram features output by the song generation model, and the target song is generated according to the target mel-spectrogram features. The real mel-spectrogram features of the target user and the song template corresponding to the target song can be effectively combined in the song generation process to effectively reduce the dependence on the data volume of the user's voice data, thereby effectively improving the song generation effect while improving the convenience of song generation.

[0208] FIG11 is a schematic structural diagram of a training device for a song generation model according to an embodiment of the present disclosure.

[0209] As shown in FIG11 , the training device 110 of the song generation model includes: a third acquisition module 1101 for acquiring a training set, the training set comes from multiple sampling users, the training set includes multiple samples, one sampling user corresponds to at least one sample, and each sample includes: the singing audio picked up when the sampling user sings a certain song and the lyrics text corresponding to the singing audio; a fourth acquisition module 1102 for acquiring a pre-built initial neural network model, the initial neural network model includes initial weight parameters and a loss function; a fifth acquisition module 1103 for acquiring the first sample from the training set, and inputting the first sample into the initial neural network model to obtain a real Mel-spectrum multi-speech signal. The true mel-spectrogram feature represents the mel-spectrogram feature of the singing audio in the first sample, and the predicted mel-spectrogram feature represents the mel-spectrogram feature predicted by the initial neural network model; the third processing module 1104 is used to calculate the error between the predicted mel-spectrogram feature and the true mel-spectrogram feature according to the loss function; the fourth processing module 1105 is used to adjust the initial weight parameters of the initial neural network model according to the error to obtain an updated neural network model; the sixth acquisition module 1106 is used to obtain subsequent samples one by one from the training set, and repeatedly input the subsequent samples into the latest neural network model until the loss function converges to obtain a trained song generation model.

[0210] In some embodiments of the present disclosure, as shown in FIG12 , FIG12 is a schematic structural diagram of a training device for a song generation model proposed in another embodiment of the present disclosure, wherein the initial neural network model includes: an initial timbre encoding sub-model, an initial text encoding sub-model, and an initial acoustic decoding sub-model; the fifth acquisition module 1103 includes: a fifth processing sub-module 11031 for performing text transcription on the lyrics text in the first sample to obtain a phoneme sequence, and aligning the singing audio pair in the first sample according to the phoneme sequence to obtain the phoneme duration; a sixth processing sub-module 11032 for performing acoustic feature extraction on the singing audio in the first sample to obtain the true mel-spectrogram feature, audio energy, and fundamental frequency trajectory of the first sample; The seventh processing submodule 11033 is used to input the phoneme sequence into the initial text encoding submodel to obtain the text feature vector of the first sample; the eighth processing submodule 11034 is used to input the real Mel spectrum feature of the first sample into the initial timbre encoding submodel to obtain the timbre feature vector of the first sample; the ninth processing submodule 11035 is used to perform time length regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain the frame-level text feature vector and the frame-level timbre feature vector; the tenth processing submodule 11036 is used to add the frame-level text feature vector, the frame-level timbre feature vector, the audio energy, and the fundamental frequency trajectory and input them into the initial acoustic decoding submodel to obtain the predicted Mel spectrum feature of the first sample.

[0211] It should be noted that the above explanation of the training method of the song generation model is also applicable to the training device of the song generation model of this embodiment, and will not be repeated here.

[0212] In this embodiment, a training set is obtained, the training set comes from multiple sampling users, the training set includes multiple samples, and each sampling user corresponds to at least one sample, each sample includes: singing audio picked up when the sampling user sings a song and lyrics text corresponding to the singing audio, a pre-built initial neural network model is obtained, the initial neural network model includes initial weight parameters and a loss function, a first sample is obtained from the training set, and the first sample is input into the initial neural network model to obtain true mel-spectrogram features and predicted mel-spectrogram features, the true mel-spectrogram features representing the mel-spectrogram features of the singing audio in the first sample, and the predicted mel-spectrogram features representing the mel-spectrogram features predicted by the initial neural network model, the error between the predicted mel-spectrogram features and the true mel-spectrogram features is calculated based on the loss function, the initial weight parameters of the initial neural network model are adjusted based on the error, and an updated neural network model is obtained, subsequent samples are obtained one by one from the training set, and the subsequent samples are repeatedly input into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model. Therefore, the error between the predicted mel-spectrogram features and the true mel-spectrogram features output by the model can be determined in real time based on the loss function during the model training process, thereby providing a reliable judgment basis for judging model convergence, thereby effectively improving the output accuracy of the song generation model.

[0213] Figure 13 shows a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present disclosure. The electronic device 12 shown in Figure 13 is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0214] 13 , electronic device 12 is implemented as a general-purpose computing device. Components of electronic device 12 may include, but are not limited to, one or more processors or processing units 16 , system memory 28 , and a bus 18 connecting various system components (including system memory 28 and processing unit 16 ).

[0215] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.

[0216] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0217] Memory 28 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer-readable storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 13 , and commonly referred to as a "hard drive").

[0218] Although not shown in FIG13 , a disk drive for reading and writing to a removable non-volatile disk (e.g., a floppy disk) and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a Compact Disc Read Only Memory (CD-ROM), a Digital Video Disc Read Only Memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data media interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present disclosure.

[0219] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0220] The electronic device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable human interaction with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). This communication can occur via an input / output (I / O) interface 22. Furthermore, the electronic device 12 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0221] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the song generation method and the song generation model training method mentioned in the above embodiments.

[0222] In order to implement the above embodiments, the present disclosure also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the song generation method and the song generation model training method proposed in the above embodiments of the present disclosure.

[0223] In order to implement the above embodiments, the present disclosure also proposes a computer program product. When the instruction processor in the computer program product is executed, it executes the song generation method and the song generation model training method proposed in the above embodiments of the present disclosure.

[0224] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0225] Those skilled in the art will understand that the various numerical numbers such as first and second involved in the present disclosure are only for the convenience of description and are not used to limit the scope of the embodiments of the present disclosure, and also indicate the order of precedence.

[0226] The at least one in the present disclosure can also be described as one or more, and the multiple can be two, three, four or more, which is not limited in the present disclosure. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", and there is no order of precedence or size between the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0227] The correspondences shown in the tables of the present disclosure can be configured or predefined. The values ​​of the information in each table are merely examples and can be configured to other values, which are not limited by the present disclosure. When configuring the correspondences between information and parameters, it is not necessarily required to configure all the correspondences shown in each table. For example, in the tables of the present disclosure, the correspondences shown in certain rows may not be configured. For another example, appropriate deformation adjustments can be made based on the above tables, such as splitting, merging, etc. The names of the parameters shown in the titles of the above tables may also adopt other names that can be understood by the communication device, and the values ​​or representations of the parameters may also adopt other values ​​or representations that can be understood by the communication device. When implementing the above tables, other data structures may also be used, such as arrays, queues, containers, stacks, linear lists, pointers, linked lists, trees, graphs, structures, classes, heaps, hash tables or hash tables, etc.

[0228] The predefined in the present disclosure may be understood as defined, predefined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.

[0229] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0230] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A song generation method, characterized in that: include: Obtain the target user's input voice audio and the unique identification number of the target song; Performing mel-spectrogram feature extraction on the speech audio to obtain the true mel-spectrogram features of the target user; Acquire a song template corresponding to the unique identification number according to the unique identification number of the target song; Inputting the target user's real mel-spectrogram features and the song template into a preset song generation model to obtain target mel-spectrogram features output by the song generation model, wherein the song generation model is trained by machine learning using a training set, the training set being from a plurality of sampled users, the training set including a plurality of samples, each sample corresponding to at least one sample, each sample including: singing audio picked up when the sampled user sings a certain song and lyrics text corresponding to the singing audio; A target song is generated according to the target mel-spectrogram features.

2. The method according to claim 1, wherein The song generation model includes: a timbre encoding sub-model, a text encoding sub-model and an acoustic decoding sub-model; the song generation model is obtained by jointly training the timbre encoding sub-model, the text encoding sub-model and the acoustic decoding sub-model using the same training set.

3. The method according to claim 2, wherein The song template includes lyrics text information and song melody information; The lyrics text information includes phoneme sequence and phoneme duration; The song melody information includes a song note sequence and a song energy sequence.

4. The method according to claim 3, wherein The step of inputting the target user's real mel-spectrogram features and the song template into a preset song generation model to obtain the target mel-spectrogram features output by the song generation model includes: Inputting the target user's real mel-spectrogram feature into the timbre encoding sub-model to obtain the target user's timbre feature vector; Inputting the phoneme sequence into the text encoding sub-model to obtain a text feature vector of the lyrics text in the song template; Performing time-length regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain a frame-level text feature vector and a frame-level timbre feature vector; The frame-level text feature vector, the frame-level timbre feature vector and the song melody information are added together and input into the acoustic decoding sub-model to obtain the target Mel spectrum feature.

5. The method according to claim 4, wherein The timbre encoding sub-model includes: a reference encoder and an autoregressive encoder; inputting the target user's real mel spectrum features into the timbre encoding sub-model to obtain the target user's timbre feature vector includes: Inputting the target user's real Mel-spectrogram features into the reference encoder to obtain the target user's timbre latent space distribution vector; Inputting the timbre latent space distribution vector into the autoregressive encoder to obtain the timbre distribution vector of the target user, wherein the timbre distribution vector is obtained by sampling the timbre latent space distribution vector by the autoregressive encoder; The timbre distribution vector is used as the timbre feature vector of the target user.

6. The method according to claim 4, wherein Duration regularization is performed on the text feature vector according to the phoneme duration to obtain a frame-level text feature vector, including: Determining an initial text encoding corresponding to each phoneme in the phoneme sequence from the text feature vector; Determining a first frame number corresponding to the phoneme according to the phoneme duration; Copying the initial text code, and performing splicing processing on the copied initial text codes of the first number of frames to obtain a target text code; The frame-level text feature vector is formed according to the multiple target text codes.

7. The method according to claim 4, wherein The timbre feature vector is time-regulated according to the phoneme duration to obtain a frame-level timbre feature vector, including: Determining a second number of frames of the speech audio according to the phoneme duration; The timbre feature vector is copied, and the copied timbre feature vectors of the second number of frames are concatenated to obtain the frame-level timbre feature vector.

8. The method according to claim 3, wherein The song template is configured by the phoneme sequence, phoneme duration, song note sequence, song energy sequence, and unique identification number of the target song, wherein the phoneme sequence and phoneme duration of the target song are determined by the song audio and song lyrics of the target song, and the song note sequence and song energy sequence of the target song are determined by the song audio.

9. The method according to claim 8, wherein The phoneme sequence includes: multiple phonemes obtained by parsing the song lyrics, and the phoneme duration includes: the first frame number occupied by each phoneme in the song audio.

10. The method according to claim 8, wherein The song energy sequence is obtained by quantizing the song energy features of the song audio, and the song note sequence is obtained by quantizing the song fundamental frequency features of the song audio.

11. The method according to claim 10, wherein The song energy feature includes: multiple energy values; the song energy sequence is formed according to multiple range coding values, and the range coding value is obtained by performing one-hot encoding processing on the energy range corresponding to the energy value.

12. The method according to claim 10, wherein The song fundamental frequency feature includes: a plurality of fundamental frequency values; the song note sequence includes a note symbol corresponding to each of the fundamental frequency values.

13. A method for training a song generation model, characterized in that: include: Obtaining a training set, the training set being from a plurality of sampled users, the training set comprising a plurality of samples, one sampled user corresponding to at least one sample, each sample comprising: singing audio picked up when the sampled user sings a song and lyrics text corresponding to the singing audio; Obtaining a pre-built initial neural network model, wherein the initial neural network model includes initial weight parameters and a loss function; Obtaining a first sample from the training set, and inputting the first sample into the initial neural network model to obtain a true mel-spectrogram feature and a predicted mel-spectrogram feature, wherein the true mel-spectrogram feature represents the mel-spectrogram feature of the singing audio in the first sample, and the predicted mel-spectrogram feature represents the mel-spectrogram feature predicted by the initial neural network model; Calculating the error between the predicted Mel spectrum feature and the true Mel spectrum feature according to the loss function; Adjusting initial weight parameters of the initial neural network model according to the error to obtain an updated neural network model; Subsequent samples are obtained one by one from the training set, and the subsequent samples are repeatedly input into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model.

14. The method according to claim 13, wherein The initial neural network model includes: an initial timbre encoding sub-model, an initial text encoding sub-model, and an initial acoustic decoding sub-model; the first sample is input into the initial neural network model to obtain a true mel spectrum feature and a predicted mel spectrum feature, including: Performing text transcription on the lyrics text in the first sample to obtain a phoneme sequence, and aligning the singing audio pairs in the first sample according to the phoneme sequence to obtain phoneme duration; Performing acoustic feature extraction on the singing audio in the first sample to obtain true mel-spectrogram features, audio energy, and fundamental frequency trajectory of the first sample; Inputting the phoneme sequence into the initial text encoding sub-model to obtain a text feature vector of the first sample; Inputting the true mel spectrum feature of the first sample into the initial timbre encoding sub-model to obtain the timbre feature vector of the first sample; Performing time-length regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain a frame-level text feature vector and a frame-level timbre feature vector; The frame-level text feature vector, the frame-level timbre feature vector, the audio energy, and the fundamental frequency trajectory are added together and input into the initial acoustic decoding sub-model to obtain the predicted Mel spectrum feature of the first sample.

15. A song generating device, characterized in that: include: A first acquisition module is used to obtain the voice audio input by the target user and the unique identification number of the target song; A first processing module is configured to extract Mel-spectrogram features from the speech audio to obtain the true Mel-spectrogram features of the target user; A second acquisition module is used to acquire a song template corresponding to the unique identification number of the target song according to the unique identification number of the target song; a second processing module, configured to input the target user's real mel-spectrogram features and the song template into a preset song generation model to obtain target mel-spectrogram features output by the song generation model, wherein the song generation model is trained by machine learning using a training set, the training set being derived from a plurality of sampled users, the training set including a plurality of samples, each sample corresponding to at least one sample, each sample including: singing audio of the sampled user singing a song and lyrics corresponding to the singing audio; A generation module is used to generate a target song according to the target mel spectrum features.

16. The device according to claim 15, characterized in that The song generation model includes: a timbre encoding sub-model, a text encoding sub-model and an acoustic decoding sub-model; the song generation model is obtained by jointly training the timbre encoding sub-model, the text encoding sub-model and the acoustic decoding sub-model using the same training set.

17. The device according to claim 16, wherein The song template includes lyrics text information and song melody information; The lyrics text information includes phoneme sequence and phoneme duration; The song melody information includes a song note sequence and a song energy sequence.

18. The device according to claim 17, wherein The second processing module includes: A first processing submodule is configured to input the target user's real mel-spectrogram feature into the timbre encoding submodel to obtain a timbre feature vector of the target user; A second processing submodule is configured to input the phoneme sequence into the text encoding submodel to obtain a text feature vector of the lyrics text in the song template; A third processing submodule is configured to perform duration regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain a frame-level text feature vector and a frame-level timbre feature vector; The fourth processing submodule is used to add the frame-level text feature vector, the frame-level timbre feature vector and the song melody information and input them into the acoustic decoding submodel to obtain the target Mel spectrum feature.

19. The device according to claim 18, wherein The first processing submodule is specifically configured to: Inputting the target user's real Mel-spectrogram features into the reference encoder to obtain the target user's timbre latent space distribution vector; Inputting the timbre latent space distribution vector into the autoregressive encoder to obtain the timbre distribution vector of the target user, wherein the timbre distribution vector is obtained by sampling the timbre latent space distribution vector by the autoregressive encoder; The timbre distribution vector is used as the timbre feature vector of the target user.

20. The device according to claim 18, wherein The third processing submodule is specifically configured to: Determining an initial text encoding corresponding to each phoneme in the phoneme sequence from the text feature vector; Determining a first frame number corresponding to the phoneme according to the phoneme duration; Copying the initial text code, and performing splicing processing on the copied initial text codes of the first number of frames to obtain a target text code; The frame-level text feature vector is formed according to the multiple target text codes.

21. The device according to claim 18, wherein The third processing submodule is further configured to: Determining a second number of frames of the speech audio according to the phoneme duration; The timbre feature vector is copied, and the copied timbre feature vectors of the second number of frames are concatenated to obtain the frame-level timbre feature vector.

22. The device according to claim 17, wherein The song template is configured by the phoneme sequence, phoneme duration, song note sequence, song energy sequence, and unique identification number of the target song, wherein the phoneme sequence and phoneme duration of the target song are determined by the song audio and song lyrics of the target song, and the song note sequence and song energy sequence of the target song are determined by the song audio.

23. The device according to claim 22, wherein The phoneme sequence includes: multiple phonemes obtained by parsing the song lyrics, and the phoneme duration includes: the first frame number occupied by each phoneme in the song audio.

24. The device according to claim 22, wherein The song energy sequence is obtained by quantizing the song energy features of the song audio, and the song note sequence is obtained by quantizing the song fundamental frequency features of the song audio.

25. The device according to claim 24, wherein The song energy feature includes: multiple energy values; the song energy sequence is formed according to multiple range coding values, and the range coding value is obtained by performing one-hot encoding processing on the energy range corresponding to the energy value.

26. The device according to claim 24, wherein The song fundamental frequency feature includes: a plurality of fundamental frequency values; the song note sequence includes a note symbol corresponding to each of the fundamental frequency values.

27. A training device for a song generation model, characterized in that: include: A third acquisition module is configured to acquire a training set, wherein the training set is from a plurality of sampled users, the training set comprising a plurality of samples, each sampled user corresponding to at least one sample, and each sample comprising: a singing audio of the sampled user singing a song and lyrics corresponding to the singing audio; A fourth acquisition module is used to acquire a pre-built initial neural network model, wherein the initial neural network model includes initial weight parameters and a loss function; a fifth acquisition module, configured to acquire a first sample from the training set and input the first sample into the initial neural network model to obtain a true mel-spectrogram feature and a predicted mel-spectrogram feature, wherein the true mel-spectrogram feature represents the mel-spectrogram feature of the singing audio in the first sample, and the predicted mel-spectrogram feature represents the mel-spectrogram feature predicted by the initial neural network model; A third processing module is configured to calculate the error between the predicted mel spectrum feature and the true mel spectrum feature according to the loss function; a fourth processing module, configured to adjust initial weight parameters of the initial neural network model according to the error to obtain an updated neural network model; The sixth acquisition module is used to obtain subsequent samples one by one from the training set, and repeatedly input the subsequent samples into the latest neural network model until the loss function converges, thereby obtaining a trained song generation model.

28. The device according to claim 27, wherein The initial neural network model includes: an initial timbre encoding sub-model, an initial text encoding sub-model, and an initial acoustic decoding sub-model; the fifth acquisition module includes: a fifth processing submodule, configured to perform text transcription on the lyrics text in the first sample to obtain a phoneme sequence, and align the singing audio pairs in the first sample according to the phoneme sequence to obtain phoneme duration; a sixth processing submodule, configured to extract acoustic features from the singing audio in the first sample to obtain true mel-spectrogram features, audio energy, and fundamental frequency trajectory of the first sample; a seventh processing submodule, configured to input the phoneme sequence into the initial text encoding submodel to obtain a text feature vector of the first sample; an eighth processing submodule, configured to input the true mel spectrum feature of the first sample into the initial timbre encoding submodel to obtain a timbre feature vector of the first sample; a ninth processing submodule, configured to perform time-length regularization on the text feature vector and the timbre feature vector according to the phoneme duration to obtain a frame-level text feature vector and a frame-level timbre feature vector; The tenth processing submodule is used to add the frame-level text feature vector, the frame-level timbre feature vector, the audio energy, and the fundamental frequency trajectory and input them into the initial acoustic decoding submodel to obtain the predicted Mel spectrum feature of the first sample.

29. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12, or the method of any one of claims 13-14.

30. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 12, or to execute the method according to any one of claims 13 to 14.

31. A computer program product, characterized in that The method comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12, or implements the steps of the method according to any one of claims 13 to 14.

Images