Method and device for training music generation model, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for training a music generation model, equipment and a storage medium. The method comprises the following steps: extracting target audio content associated with singing content from a music sample; based on the target audio content, annotation information is generated, and the annotation information comprises first text content corresponding to the target audio content and first melody information of the target audio content; generating a first token sequence corresponding to the first control signal based on the text content and the first melody information; a plurality of training sequences are constructed based on the plurality of control signals, the control signals comprise a first control signal, and the plurality of training sequences correspond to different control signal combinations; and training the music generation model by using the plurality of training sequences. In this way, according to the embodiment of the invention, the training cost of the model can be reduced.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for training music generation models. Background Technology

[0002] With the development of machine learning technology, machine learning models can now be used to perform tasks in various application environments. Singing Voice Synthesis (SVS) is a computer technology that attempts to simulate human singing. SVS can be seen as a special branch of Text-to-Speech (TTS). In synthesizing songs using SVS, it's not only necessary to maintain the intelligibility of the language, but also to replicate musical features such as timbre, pitch, duration, and singing style as accurately as possible. However, SVS technology still has some problems that affect the quality of the synthesized songs. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for training a music generation model is provided. The method includes: extracting target audio content associated with singing content from music samples; generating annotation information based on the target audio content, the annotation information including first text content corresponding to the target audio content and first melody information of the target audio content; generating a first token sequence corresponding to a first control signal based on the text content and the first melody information; constructing multiple training sequences based on multiple control signals, the control signals including the first control signal, the multiple training sequences corresponding to different combinations of control signals; and training the music generation model using the multiple training sequences.

[0004] In a second aspect of this disclosure, an apparatus for training a music generation model is provided. The apparatus includes: a content extraction module configured to extract target audio content associated with singing content from music samples; a first generation module configured to generate annotation information based on the target audio content, the annotation information including first text content corresponding to the target audio content and first melody information of the target audio content; a second generation module configured to generate a first token sequence corresponding to a first control signal based on the text content and the first melody information; a sequence construction module configured to construct multiple training sequences based on multiple control signals, the control signals including the first control signal, the multiple training sequences corresponding to different combinations of control signals; and a model training module configured to train a music generation model using the multiple training sequences.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A schematic diagram illustrating a model training process according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram illustrating a model reasoning process according to some embodiments of the present disclosure is shown;

[0012] Figure 4A and Figure 4B A schematic diagram illustrating the determination of style features according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A flowchart illustrating an example process for training a music generation model according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A block diagram of an apparatus for training a music generation model according to some embodiments of the present disclosure is shown; and

[0015] Figure 7 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0018] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0019] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0020] It is understood that the embodiments disclosed herein involve model training and inference, and the data involved (including but not limited to the data itself, the acquisition or use of the data) comply with the requirements of relevant laws, regulations and related provisions.

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0023] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0026] As used herein, a “unit,” “operation unit,” or “subunit” can consist of any suitable machine learning model or network. As used herein, a set of elements or similar expressions can include one or more such elements. For example, “a set of convolutional units” can include one or more convolutional units.

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, model 130-1 with pre-training parameter values ​​and model 130-2 with post-training parameter values ​​can be collectively referred to as model 130 or referred to individually. Model 130 can be implemented or included in electronic device 140.

[0028] exist Figure 1 In environment 100, it is desired to train and use a machine learning model (i.e., model 130) configured for various application environments. For example, if the model is a speech synthesis model, it can generate speech corresponding to the text based on reference speech and text input by the user, or edit the speech information input by the user. For example, if model 130 is a song synthesis model, it can generate corresponding song audio based on sheet music information and singing information input by the user.

[0029] like Figure 1 As shown, environment 100 includes electronic device 140. Electronic device 140 may contain a model training system and / or a model application system. Figure 1 The upper part illustrates the model training phase, and the lower part illustrates the model application phase. Before training, the parameter values ​​of model 130 can have initial values ​​or pre-trained parameter values ​​obtained through a pre-training process. Model 130-1 can be trained via forward and backward propagation, during which the parameter values ​​of model 130-1 can be updated and adjusted. After training, model 130-2 is obtained. At this point, the parameter values ​​of model 130-2 have been updated, and based on the updated parameter values, model 130-2 can be used to implement the song synthesis task in the model application phase.

[0030] During the model training phase, model 130 can be trained using a model training system based on a training sample set 110 comprising multiple training samples 112. Each training sample 112 can involve a binary format. For example, for a song synthesis task, training sample 112 can include training input 120 and training output for the song synthesis task. The training input in the song synthesis task can, for example, include reference audio and sheet music. Training sample 112, including model input 120 and model output 122, can be used to train model 130. Specifically, the training process can be performed iteratively using a large number of training samples. After training is complete, model 130 can include knowledge about the task to be processed. During the model application phase, model 130 (at this point, model 130 has the trained parameter values) can be used to perform the corresponding task. For example, model input 142 from the song synthesis task can be received, and corresponding model output 144 can be output.

[0031] exist Figure 1 In this context, electronic device 140 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices may involve any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0032] It should be understood that Figure 1 The components and arrangements shown in environment 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. Implementations of this disclosure are not limited in this respect.

[0033] As briefly mentioned earlier, song synthesis technology is used to generate songs based on input text and musical notes. Song synthesis technology includes splicing synthesis technology and artificial intelligence (AI) synthesis technology. AI synthesis technology uses machine learning models to learn musical characteristics of human voice samples, such as timbre, pitch, phoneme duration, and singing style.

[0034] Currently, the mainstream song synthesis technology is represented by the DiffSinger model. DiffSinger is a song synthesis method based on a diffusion model, which enhances the control over the singing voice by adding parameters such as pitch, dynamics, gender, and energy to generate songs that meet specific requirements.

[0035] However, the current DiffSinger model is based on phoneme modeling. This requires a large amount of manually annotated phoneme data, increasing modeling costs. Furthermore, the manual annotations in the phoneme data can disrupt the regularity of note durations in the musical score. Therefore, an additional phoneme duration prediction model is needed to predict the duration of each phoneme, further increasing the cost of training the model.

[0036] Embodiments of this disclosure propose a scheme for training a music generation model. According to various embodiments of this disclosure, target audio content associated with singing content can be extracted from music samples. Further, annotation information can be generated based on the target audio content, the annotation information including first text content corresponding to the target audio content and first melody information of the target audio content.

[0037] Additionally, a first token sequence corresponding to the first control signal can be generated based on the text content and the first melody information. Furthermore, multiple training sequences can be constructed based on multiple control signals, including the first control signal, with each training sequence corresponding to a different combination of control signals. Accordingly, a music generation model can be trained using these multiple training sequences.

[0038] In this way, on the one hand, by constructing training sequences from text and melody content, the embodiments of this disclosure can reduce the annotation cost of the training process. On the other hand, by constructing training sequences through combinations of different control signals, the embodiments of this disclosure can further improve the performance of the music generation model to adapt to different generation scenarios.

[0039] Specific embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings.

[0040] Training of music generation models

[0041] Figure 2A schematic diagram 200 illustrates a model training process according to some embodiments of the present disclosure. For example... Figure 2 As shown, during the training of the music generation model, the electronic device 140 can acquire music samples 205 and can use the sound source separation module 210 to extract human voice audio 215 from the music samples 205, that is, the target audio content associated with the singing content.

[0042] Furthermore, the electronic device 140 can generate corresponding annotation information 230 based on the human voice audio 215. Specifically, as shown in the figure, the electronic device 140 can use the text recognition module 220 to process the human voice audio 215 to generate first text content. Additionally, the electronic device 140 can also use the melody recognition module 225 to process the human voice audio 215 to determine first melody information.

[0043] In some embodiments, the melody recognition module 225 may include, for example, a vocal MIDI (Musical Instrument Digital Interface) module to recognize a set of notes corresponding to the vocal audio 215.

[0044] Furthermore, the electronic device 140 can generate a first token sequence 240 based on the first text content and the first melody information included in the annotation information 230. Specifically, as shown in the figure, the electronic device 140 can use the discrete encoder 235 to generate a first token sequence corresponding to the annotation information 230, also known as a text pitch token sequence 240.

[0045] In some embodiments, the electronic device 110 can determine the first time information corresponding to each lyric element in the first text content, that is, the time of appearance and duration of each lyric element. Further, the electronic device 110 can determine a first sequence portion corresponding to the text content, i.e., a text token sequence, based on the first text content and the first time information corresponding to the first text content.

[0046] In some embodiments, the text token sequence may indicate a first text content and the time distribution corresponding to each lyric element in the first text content. The time distribution of each lyric element in the first text content may, for example, indicate the occurrence time and duration of each lyric element.

[0047] In some embodiments, the electronic device 110 may also construct a second sequence portion, namely, a pitch token sequence, based on the first melody information and the second time information corresponding to the first melody information.

[0048] In some embodiments, the pitch token sequence may indicate a set of melodic elements corresponding to the first melody information and the time distribution corresponding to the melodic elements in the set of melodic elements.

[0049] The first and second sequence portions are used for text pitch token sequence 240. Figure 2 As an example, the annotation information 230 can be represented as [word1, w_dur1, word2, w_dur2, ..., note1, n_dur1, note2, n_dur2, ...]. Therefore, the electronic device 110 can construct a text token sequence based on the parts of the annotation information 230 corresponding to the lyrics, and construct a pitch token sequence based on the parts of the annotation information 230 corresponding to the melody, thereby obtaining a text pitch token sequence 240.

[0050] In some embodiments, the electronic device 140 may also improve the stability of the model based on classifier-free guidance (CFG).

[0051] Specifically, the electronic device 140 can determine multiple control signals. For example, the electronic device 140 can use the text pitch token sequence 240 (i.e., the first token sequence) as the first control signal. Additionally, the electronic device 140 can also use the style feature corresponding to the music sample 205 as the second control signal. Such a style feature can, for example, characterize the singing style of the music sample 205.

[0052] Furthermore, the electronic device 140 can construct multiple training sequences based on multiple control signals (e.g., a first control signal and a second control signal), wherein the multiple training sequences correspond to different combinations of control signals.

[0053] As an example, such as Figure 2 As shown, the electronic device 140 can combine a first control signal (i.e., a text pitch token), a second control signal (i.e., a style feature), and audio features 245 to construct a first training sequence.

[0054] As another example, such as Figure 2 As shown, the electronic device 140 can combine the first control signal (i.e., the text pitch token) and the audio feature 245 to construct a second training sequence. That is, the second training sequence may not include the second control signal.

[0055] As yet another example, such as Figure 2 As shown, the electronic device 140 can combine the second control signal (i.e., style feature) and audio feature 245 to construct a third training sequence. That is, the third training sequence may not include the first control signal.

[0056] As yet another example, such as Figure 2As shown, the electronic device 140 can construct a fourth training sequence based on the audio features 245. That is, the fourth training sequence may not include the first control signal and the second control signal.

[0057] It should be understood that the training sequence mentioned above does not include specific control signals, which may mean that preset placeholders are used to replace the sequence part corresponding to the control signals.

[0058] Specifically, the audio feature 245 can be an audio token generated by processing the feature representation 250 of the human voice audio 215 using a discrete encoder 255.

[0059] Furthermore, the electronic device 140 can utilize the constructed multiple training sequences to train a music generation model. In some embodiments, such as Figure 2 As shown, the music generation model may include, for example, a diffusion model unit 260.

[0060] During the processing of the diffusion model unit 260, the diffusion model unit 260 can perform noise addition and noise reduction processes on the audio features 240 in the target training sequence based on the received target training sequence.

[0061] During the training of the diffusion model unit 260, the electronic device 140 can adjust the model parameters of the diffusion model unit 260 based on the loss function of the diffusion model.

[0062] Specifically, the electronic device 140 can acquire intermediate audio features output by the diffusion model unit 260. Further, the electronic device 140 can train a music generation model based on a comparison of corresponding portions of the intermediate audio features with reference audio features. As an example, the reference audio features may include audio features 245 corresponding to the human voice audio 215.

[0063] In this way, on the one hand, by constructing training sequences from text and melody content, the embodiments of this disclosure can reduce the annotation cost of the training process. On the other hand, by constructing training sequences through combinations of different control signals, the embodiments of this disclosure can further improve the performance of the music generation model to adapt to different generation scenarios.

[0064] Inference of music generation models

[0065] Figure 3 A schematic diagram 300 illustrates a model reasoning process according to some embodiments of the present disclosure. For example... Figure 3 As shown, during the inference process of the music generation model, the electronic device 140 can acquire the score information 305. As an example, the score information 305 may be a musicXML file.

[0066] Furthermore, the electronic device 140 can extract target content 310 based on the musical score information 305. Target content 310 may include second text content and second melody information. For example, the second text content may correspond to the lyrics in the musical score information 305. The second melody information may indicate a set of melodic elements indicated by the musical score information 305.

[0067] Furthermore, the electronic device 140 can utilize the encoder 310 to process the second text content and the second melody information to construct a text pitch token sequence 320 (also known as a second token sequence).

[0068] Furthermore, the electronic device 140 can construct a first input sequence for the music generation model based on the text pitch token sequence 320. For example, the electronic device 140 can construct the first input sequence by combining the text pitch token sequence 320 with preset noise (e.g., Gaussian noise 330).

[0069] In some embodiments, the first input sequence may also indicate style features corresponding to a target singing style. In some embodiments, such a target singing style may be a specific singer, a specific singing type, etc. As an example, style features may include the timbre features of the musical content to be generated.

[0070] Furthermore, such as Figure 3 As shown, the electronic device 140 can use a trained diffusion model unit 325 to process the first input sequence to generate audio features 335 (also referred to as first audio features). Further, the electronic device 110 can use a decoder 340 and a vocoder 345 to process the audio features 335 to decode the audio features 335 into target music content.

[0071] In some embodiments, in order to improve control over the generated music content, the electronic device 140 may also generate the final target music content based on a mixture of conditional and unconditional inputs.

[0072] Specifically, the electronic device 110 can also construct a second input sequence independent of the second token sequence (i.e., the text pitch token sequence). For example, the electronic device 110 can construct a second input sequence that does not include control signals.

[0073] Accordingly, the electronic device 140 can use a trained music generation model to process the second input sequence to generate second audio features. Further, the electronic device 140 can determine target audio features based on the first and second audio features, and can use the decoder 340 and vocoder 345 to decode the target audio features, thereby generating target music content 350.

[0074] In some embodiments, the target audio features may be represented, for example, as a weighted sum of a first audio feature and a second audio feature. Accordingly, the electronic device 140 may also control the clarity of pronunciation of the generated target music content 350 through weighting coefficients. For example, the greater the weight of the first audio feature, the higher the clarity of pronunciation.

[0075] In this way, the embodiments of this disclosure can utilize a diffusion model and generate corresponding music content based on musical score information, thereby improving the efficiency of music content generation.

[0076] Control of musical style

[0077] As mentioned above, in both the training and inference phases of a music generation model, the input sequences of the model can include style features.

[0078] In some embodiments, the style features may be, for example, preset style features corresponding to the input singing tags. For instance, a set of preset style features corresponding to a set of preset singing tags can be pre-constructed, and the corresponding preset style features can be obtained based on the singing tags specified by the user.

[0079] In some embodiments, such as referring to Figure 4A and Figure 4B The style features discussed can also be generated by processing the corresponding audio content using a style coding model.

[0080] Figure 4A The training process 400A of a style coding model according to some embodiments of the present disclosure is illustrated. Considering that the singing style of music can be represented by the residual between the fundamental frequency and the pitch of the notes, during the training of the style coding model, the electronic device 140 can acquire a pitch residual sequence 405 of the training audio content. The pitch reference sequence 405 can indicate the difference between the fundamental frequency and the pitch of a set of notes. Further, the electronic device 110 can utilize an encoder 410 and a quantizer 415 to convert this into a set of discrete features.

[0081] Accordingly, the electronic device 140 can also use the decoder 420 to decode the set of discrete features into a residual prediction sequence 425, and can train the encoder 410, quantizer 415 and decoder 420 based on the difference between the residual prediction sequence 425 and the pitch residual sequence 405.

[0082] Figure 4B The process of generating style features using a style coding model is further illustrated 400B. For example... Figure 4BAs shown, the electronic device 140 can acquire the prompt audio content 430. During the inference phase of the music generation model, the prompt audio content 430 can include any appropriate audio content input by the user, such as an audio clip hummed by the user, an audio clip of a specific singer's performance, etc.

[0083] Furthermore, the electronic device 140 can determine the pitch residual sequence 435 of the prompt audio content 430, and can use the trained encoder 410 and quantizer 415 to convert the pitch residual sequence 435 into a set of discrete features for use as the input sequence for constructing the diffusion model unit.

[0084] In this way, embodiments of the present disclosure can achieve more flexible control over music style, thereby improving the quality of the generated music content.

[0085] Example process

[0086] Figure 5 A flowchart of a process 500 for training a music generation model according to some embodiments of the present disclosure is shown. Process 500 can be implemented at an electronic device 140.

[0087] In box 510, electronic device 140 extracts target audio content associated with the singing content from the music sample.

[0088] In box 520, electronic device 140 generates annotation information based on target audio content. The annotation information includes first text content corresponding to the target audio content and first melody information of the target audio content.

[0089] In box 530, electronic device 140 generates a first token sequence corresponding to the first control signal based on the text content and the first melody information.

[0090] In box 540, electronic device 140 constructs multiple training sequences based on multiple control signals, the control signals including a first control signal, and the multiple training sequences corresponding to different combinations of control signals.

[0091] In box 550, electronic device 140 uses multiple training sequences to train a music generation model.

[0092] In some embodiments, generating annotation information includes: processing the target audio content using a text recognition model to determine first text content; and / or processing the target audio content using a melody recognition model to determine first melody information.

[0093] In some embodiments, generating a first token sequence corresponding to a first control signal based on first text content and first melody information includes: constructing a first sequence portion corresponding to the first text content based on the first text content and first time information corresponding to the first text content; constructing a second sequence portion corresponding to the first melody information based on the first melody information and second time information corresponding to the first melody information; and constructing a first token sequence based on the first sequence portion and the second sequence portion.

[0094] In some embodiments, the first sequence portion indicates multiple lyric elements and the time distribution corresponding to the multiple lyric elements, and / or the second sequence portion indicates a set of melodic elements and the time distribution corresponding to the set of melodic elements.

[0095] In some embodiments, training a music generation model using multiple training sequences includes: determining reference audio features of target audio content; providing the music generation model with the reference audio features and a target training sequence from the multiple training sequences; having the music generation model perform noise addition and denoising processes on the audio features based on the target training sequence to generate intermediate audio features; and training the music generation model based on a comparison of corresponding parts of the intermediate audio features and the reference audio features.

[0096] In some embodiments, the plurality of control signals further include a second control signal, which is used to characterize the style features corresponding to the music sample.

[0097] In some embodiments, process 500 further includes: determining second text content and second melody information based on musical score information; constructing a second token sequence based on the second text content and second melody information; constructing a first input sequence for a music generation model based on the second token sequence; and processing the first input sequence using a trained music generation model to generate target music content.

[0098] In some embodiments, the first input sequence further indicates style features corresponding to the target singing style.

[0099] In some embodiments, style features include: preset style features corresponding to the input singing tag; and / or style features generated by processing the cue audio content using a style coding model.

[0100] In some embodiments, the style coding model generates style features based on a pitch residual sequence of cue audio content, the pitch residual sequence indicating the difference between the pitch and fundamental frequency of a set of notes in the cue audio content.

[0101] In some embodiments, processing a first input sequence using a trained music generation model to generate target music content includes: constructing a second input sequence independent of the second token sequence; processing the first input sequence and the second input sequence using the trained music generation model to generate a first audio feature and a second audio feature; determining a target audio feature based on the first audio feature and the second audio feature; and generating target music content by decoding the target audio feature.

[0102] In some embodiments, the target audio features are a weighted sum of the first audio features and the second audio features.

[0103] In some embodiments, the music generation model includes a diffusion model unit.

[0104] Example devices and equipment

[0105] Figure 6 A schematic structural block diagram of an apparatus 600 for training a music generation model according to certain embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in an electronic device. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0106] like Figure 6 As shown, the device 600 includes a content extraction module 610 configured to extract target audio content associated with singing content from music samples; a first generation module 620 configured to generate annotation information based on the target audio content, the annotation information including first text content corresponding to the target audio content and first melody information of the target audio content; a second generation module 630 configured to generate a first token sequence corresponding to a first control signal based on the text content and the first melody information; a sequence construction module 640 configured to construct multiple training sequences based on multiple control signals, the control signals including the first control signal, the multiple training sequences corresponding to different combinations of control signals; and a model training module 650 configured to train a music generation model using the multiple training sequences.

[0107] In some embodiments, the first generation module 620 is further configured to: process the target audio content using a text recognition model to determine first text content; and / or process the target audio content using a melody recognition model to determine first melody information.

[0108] In some embodiments, the second generation module 630 is further configured to: construct a first sequence portion corresponding to the first text content based on the first text content and the first time information corresponding to the first text content; construct a second sequence portion corresponding to the first melody information based on the first melody information and the second time information corresponding to the first melody information; and construct a first token sequence based on the first sequence portion and the second sequence portion.

[0109] In some embodiments, the first sequence portion indicates multiple lyric elements and the time distribution corresponding to the multiple lyric elements, and / or the second sequence portion indicates a set of melodic elements and the time distribution corresponding to the set of melodic elements.

[0110] In some embodiments, the model training module 650 is further configured to: determine reference audio features of the target audio content; provide the reference audio features and a target training sequence from a plurality of training sequences to the music generation model; have the music generation model perform a noise addition and a noise reduction process on the audio features based on the target training sequence to generate intermediate audio features; and train the music generation model based on a comparison of corresponding parts of the intermediate audio features and the reference audio features.

[0111] In some embodiments, the plurality of control signals further include a second control signal, which is used to characterize the style features corresponding to the music sample.

[0112] In some embodiments, the device 600 further includes a content generation module configured to: determine second text content and second melody information based on musical score information; construct a second token sequence based on the second text content and second melody information; construct a first input sequence for a music generation model based on the second token sequence; and process the first input sequence using a trained music generation model to generate target music content.

[0113] In some embodiments, the first input sequence further indicates style features corresponding to the target singing style.

[0114] In some embodiments, style features include: preset style features corresponding to the input singing tag; and / or style features generated by processing the cue audio content using a style coding model.

[0115] In some embodiments, the style coding model generates style features based on a pitch residual sequence of cue audio content, the pitch residual sequence indicating the difference between the pitch and fundamental frequency of a set of notes in the cue audio content.

[0116] In some embodiments, the content generation module is further configured to: construct a second input sequence independent of the second token sequence; process the first input sequence and the second input sequence respectively using a trained music generation model to generate a first audio feature and a second audio feature; determine a target audio feature based on the first audio feature and the second audio feature; and generate target music content by decoding the target audio feature.

[0117] In some embodiments, the target audio features are a weighted sum of the first audio features and the second audio features.

[0118] In some embodiments, the music generation model includes a diffusion model unit.

[0119] Figure 7 A block diagram is shown illustrating an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 140 electronic devices.

[0120] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0121] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0122] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0123] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0124] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0125] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0126] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0128] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0130] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for training a music generation model, comprising: extracting, from a music sample, target audio content associated with singing content; generating, based on the target audio content, annotation information comprising first text content corresponding to the target audio content and first melody information of the target audio content; generating, based on the text content and the first melody information, a first token sequence corresponding to a first control signal; constructing, based on a plurality of control signals comprising the first control signal, a plurality of training sequences corresponding to different combinations of control signals; and training, using the plurality of training sequences, a music generation model.

2. The method of claim 1, wherein generating annotation information comprises: processing the target audio content using a text recognition model to determine the first text content; and / or processing the target audio content using a melody recognition model to determine the first melody information.

3. The method of claim 2, wherein generating, based on the first text content and the first melody information, a first token sequence corresponding to a first control signal comprises: constructing, based on the first text content and first time information corresponding to the first text content, a first sequence part corresponding to the first text content; constructing, based on the first melody information and second time information corresponding to the first melody information, a second sequence part corresponding to the first melody information; and constructing, based on the first sequence part and the second sequence part, the first token sequence.

4. The method of claim 3, wherein: the first sequence part indicates a plurality of lyric elements and a time distribution corresponding to the plurality of lyric elements, and / or the second sequence part indicates a set of melody elements and a time distribution corresponding to the set of melody elements.

5. The method of claim 1, wherein training, using the plurality of training sequences, a music generation model comprises: determining reference audio features of the target audio content; providing, to the music generation model, the reference audio features and a target training sequence of the plurality of training sequences; applying, by the music generation model, a noise adding process and a noise removing process to the audio features based on the target training sequence to generate intermediate audio features; and training, based on a comparison of the intermediate audio features and corresponding parts of the reference audio features, the music generation model.

6. The method of claim 1, wherein the plurality of control signals further comprises a second control signal for characterizing style features corresponding to the music sample.

7. The method of claim 1, further comprising: determining, based on score information, second text content and second melody information; constructing, based on the second text content and the second melody information, a second token sequence; constructing, based on the second token sequence, a first input sequence for the music generation model; processing, using the trained music generation model, the first input sequence to generate target music content. ​ ​ ​ ​ 8. The method of claim 7, wherein the first input sequence further indicates a style feature corresponding to a target singing style.

9. The method of claim 6 or 8, wherein the style feature comprises: a preset style feature corresponding to an input singing label; and / or a style feature generated by processing prompt audio content with a style encoding model.

10. The method of claim 9, wherein the style encoding model generates the style feature based on a pitch residual sequence of the prompt audio content, the pitch residual sequence indicating a difference between a pitch of a set of notes of the prompt audio content and a base frequency.

11. The method of claim 7, wherein processing the first input sequence with the trained music generation model to generate target music content comprises: constructing a second input sequence independent of the second token sequence; processing the first input sequence and the second input sequence with the trained music generation model respectively to generate first audio features and second audio features; determining target audio features based on the first audio features and the second audio features; generating the target music content by decoding the target audio features.

12. The method of claim 11, wherein the target audio features are a weighted sum of the first audio features and the second audio features.

13. The method of claim 1, wherein the music generation model comprises a diffusion model unit.

14. An apparatus for training a music generation model, comprising: a content extraction module configured to extract target audio content associated with singing content from a music sample; a first generation module configured to generate annotation information based on the target audio content, the annotation information comprising first text content corresponding to the target audio content and first melody information of the target audio content; a second generation module configured to generate a first token sequence corresponding to a first control signal based on the text content and the first melody information; a sequence construction module configured to construct a plurality of training sequences based on a plurality of control signals, the control signals comprising the first control signal, the plurality of training sequences corresponding to different combinations of control signals; and a model training module configured to train a music generation model with the plurality of training sequences.

15. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method of any one of claims 1-13.

16. A computer-readable storage medium having stored thereon a computer program executable by a processor to implement the method of any one of claims 1-13. ​ ​ ​

Images