Music generation method, apparatus, device, storage medium, and program product

Abstract

The present disclosure relates to a music generation method, device, equipment, storage medium and program product, wherein the music generation method comprises: obtaining text information for describing target music; inputting the text information into a target first model to encode the text information, obtaining a first text sequence, and predicting a hidden vector sequence for identifying music based on the first text sequence; inputting the text information and the hidden vector sequence into a second model to encode the text information, obtaining a second text sequence, and predicting a hidden vector sequence for identifying noise based on the second text sequence and the hidden vector sequence for identifying music; denoising the hidden vector sequence for identifying music based on the hidden vector sequence for identifying noise, obtaining a denoised hidden vector sequence, and generating target music based on the denoised hidden vector sequence. The method generates music with high structure, and optimizes the generated music by denoising, so that the generated music has high music effect and structure.

Description

Technical Field

[0001] This disclosure relates to the field of music artificial intelligence, and in particular to music generation methods, devices, equipment, storage media, and program products. Background Technology

[0002] With the development of music generation technology, artificial intelligence algorithms can be used to generate music that matches the text description based on text information.

[0003] In related technologies, the music generator model (MusicGen) can generate music that conforms to the text information and has a complete structure, but the music quality of the generated music is relatively rough. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a music generation method, apparatus, device, storage medium, and program product.

[0005] According to some embodiments of this disclosure, a music generation method is provided, comprising: acquiring text information describing target music; inputting the text information into a first target model to encode the text information to obtain a first text sequence, and predicting a latent vector sequence identifying the music based on the first text sequence; inputting the text information and the latent vector sequence into a second model to encode the text information to obtain a second text sequence, and predicting a latent vector sequence identifying noise based on the second text sequence and the latent vector sequence identifying the music, wherein the noise is noise included in the latent vector sequence identifying the music; denoising the latent vector sequence identifying the music based on the latent vector sequence identifying the noise to obtain a denoised latent vector sequence, and generating the target music based on the denoised latent vector sequence.

[0006] According to some embodiments of this disclosure, the first model is trained as follows: music samples and text information describing the music samples are acquired; the text information describing the music samples is encoded to obtain a third text sequence, and each frame in the music samples is encoded to obtain a codebook sequence of the music samples, wherein each frame corresponds to K codebooks, and the length of the codebook sequence of the music samples is the number of frames in the music samples; the third text sequence is input into the first model to be trained to obtain a latent vector sequence identifying the music; based on the latent vector sequence identifying the music, the vector after K normalization calculations is performed and the K codebooks are used to determine the cross-entropy loss, and the first model to be trained is adjusted; the above process is repeated until the cross-entropy loss satisfies a first condition, thus obtaining the first model.

[0007] According to some embodiments of this disclosure, the music samples include multiple samples, and different samples correspond to different duration ranges; the first model includes multiple models, and different first models are trained based on music samples with different duration ranges; the text information describing the target music includes the target duration, and the target first model is trained based on music samples with the duration range corresponding to the target duration.

[0008] According to some embodiments of this disclosure, the second model is trained as follows: obtaining text information describing the music sample and a vector sequence identifying the music obtained based on a third text sequence; encoding the text information describing the music sample to obtain a fourth text sequence, and adding noise to the latent vector sequence obtained based on the third text sequence to obtain a latent vector sequence identifying the noise-added music; inputting the fourth text sequence and the latent vector sequence identifying the noise-added music into the second model to be trained to obtain the predicted noise; adjusting the second model to be trained based on the predicted noise until the predicted noise matches the added noise, thus obtaining the second model.

[0009] According to some embodiments of this disclosure, the step of denoising the latent vector sequence of the identified music based on the latent vector sequence of the identified noise to obtain a denoised latent vector sequence includes: performing a first denoising on the latent vector sequence of the identified music based on the latent vector sequence of the identified noise to obtain a first denoised latent vector sequence; inputting the first denoised latent vector sequence and the text information into a second model to denoise the denoised latent vector sequence to obtain a second denoised latent vector sequence; and using the second denoised latent vector sequence as the denoised latent vector sequence.

[0010] According to some embodiments of this disclosure, a music generation apparatus is provided, comprising: an acquisition module configured to acquire text information describing target music; a generation module configured to input the text information into a first target model to encode the text information to obtain a first text sequence, and predict a latent vector sequence identifying the music based on the first text sequence; an optimization module configured to input the text information and the latent vector sequence into a second model to encode the text information to obtain a second text sequence, and predict a latent vector sequence identifying noise based on the second text sequence and the latent vector sequence identifying the music, wherein the noise is noise included in the latent vector sequence identifying the music; and denoising the latent vector sequence identifying the music based on the latent vector sequence identifying the noise to obtain a denoised latent vector sequence, and generating the target music based on the denoised latent vector sequence.

[0011] According to some embodiments of this disclosure, the first model is trained as follows: music samples and text information describing the music samples are acquired; the text information describing the music samples is encoded to obtain a third text sequence, and each frame in the music samples is encoded to obtain a codebook sequence of the music samples, wherein each frame corresponds to K codebooks, and the length of the codebook sequence of the music samples is the number of frames in the music samples; the third text sequence is input into the first model to be trained to obtain a latent vector sequence identifying the music; based on the latent vector sequence identifying the music, the vector after K normalization calculations is performed and the K codebooks are used to determine the cross-entropy loss, and the first model to be trained is adjusted; the above process is repeated until the cross-entropy loss satisfies a first condition, thus obtaining the first model.

[0012] According to some embodiments of this disclosure, the music samples include multiple samples, and different samples correspond to different duration ranges; the first model includes multiple models, and different first models are trained based on music samples with different duration ranges; the text information describing the target music includes the target duration, and the target first model is trained based on music samples with the duration range corresponding to the target duration.

[0013] According to some embodiments of this disclosure, the second model is trained as follows: obtaining text information describing the music sample and a vector sequence identifying the music obtained based on a third text sequence; encoding the text information describing the music sample to obtain a fourth text sequence, and adding noise to the latent vector sequence obtained based on the third text sequence to obtain a latent vector sequence identifying the noise-added music; inputting the fourth text sequence and the latent vector sequence identifying the noise-added music into the second model to be trained to obtain the predicted noise; adjusting the second model to be trained based on the predicted noise until the predicted noise matches the added noise, thus obtaining the second model.

[0014] According to some embodiments of this disclosure, the optimization module denoises the latent vector sequence of the identified music based on the latent vector sequence of the identified noise in the following manner to obtain a denoised latent vector sequence: the latent vector sequence of the identified music is denoised for the first time based on the latent vector sequence of the identified noise to obtain a first denoised latent vector sequence; the first denoised latent vector sequence and the text information are input into a second model to denoise the denoised latent vector sequence to obtain a second denoised latent vector sequence; the second denoised latent vector sequence is used as the denoised latent vector sequence.

[0015] According to some embodiments of this disclosure, another music generation device is provided, including a processor; a memory for storing processor-executable computer programs or instructions; wherein the processor is configured to execute the computer programs or instructions to implement the music generation method described in some embodiments of this disclosure.

[0016] According to some embodiments of the present disclosure, a storage medium is provided, including a computer program or instructions stored in the storage medium, which, when executed by a music generation processor, enables music generation to perform the music generation method described in some embodiments of the present disclosure.

[0017] According to some embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the music generation method described in some embodiments of this disclosure.

[0018] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by taking the process of generating a sequence of identified music based on text information as the first stage, and after obtaining the latent vector sequence of identified music with complete structure, a second stage is added to denoise and optimize the latent vector sequence of identified music using a second model, thereby improving the musical effect of the target music. This method generates a latent vector sequence of identified music capable of identifying music with complete structure in the first stage, and then denoises and optimizes the latent vector sequence of identified music in the second stage, thus balancing the structural integrity and musical effect of the generated music and improving the user experience. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a flowchart illustrating a music generation method according to some embodiments of the present disclosure.

[0021] Figure 2 This is a flowchart illustrating how a first model learns the structural features of music according to some embodiments of the present disclosure.

[0022] Figure 3 This is a flowchart illustrating a method for generating music of a target duration according to some embodiments of the present disclosure.

[0023] Figure 4 This is a flowchart illustrating a method for enabling a model to predict noise in a sequence of latent vectors identifying music, according to some embodiments of this disclosure.

[0024] Figure 5This is a flowchart illustrating a method for improving the musical effect of a target music through multiple noise predictions, according to some embodiments of the present disclosure.

[0025] Figure 6 This is a block diagram of a music generation apparatus according to some embodiments of the present disclosure.

[0026] Figure 7 This is a block diagram illustrating an apparatus for music generation according to some embodiments of the present disclosure.

[0027] Figure 8 This is a block diagram of an apparatus for music generation, illustrated according to some embodiments disclosed in a book. Detailed Implementation

[0028] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0029] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] This disclosure provides some embodiments of a music generation method that is applied to scenarios where music is generated based on descriptive information.

[0031] In related technologies, the development of music generation mainly involves methods based on rules, probabilistic models, and deep learning. Music generation methods are primarily divided into two categories: generating musical notation and generating waveforms. Musical notation is discrete data, closely resembling musical scores. Common deep learning methods include Convolutional Neural Networks (CNNs), Recurrent Neural Networks / Long Short-Term Memory (RNNs / LSTMs), Transformer models, and Diffusion models. These methods typically employ data processing techniques suitable for their respective models to convert raw Musical Instrument Digital Interface (MIDI) data into training data. The MusicGen model can convert music into a discrete codebook representation. Through an autoregressive transducer model, it learns the structural form of music from the discrete codebook, enabling the generation of new music based on descriptive information. Furthermore, music generated by the MusicGen model has good structure. However, because converting music into a codebook introduces errors, music generated by the MusicGen model suffers from a rough musical quality. Methods for generating music also include fine-tuned large language models such as LLaMa, which are fine-tuned using music text data to generate music text, or, by quantizing the music to obtain multi-level codes to represent music frames, thereby using large language models to predict the sequence of music frames in order.

[0032] In view of this, some embodiments of this disclosure provide a music generation method. This method generates basic music information with good structure based on descriptive information using a music generation model, and optimizes the music effect of the basic music information generated by the music generation model by denoising the basic music information based on the descriptive information, thereby achieving music with both good structure and good musical effect.

[0033] For ease of description, the embodiments of this disclosure will be described using textual information. The music generation model that generates basic music information based on the descriptive information will be referred to as the first model, and the model that performs noise reduction and optimization on the generated basic music information based on the descriptive information will be referred to as the second model. The first model may, for example, be an autoregressive converter model consisting of 24 converter layers, while the second model may, for example, be a prediction model combining a diffusion model and a converter model.

[0034] Figure 1 This is a flowchart illustrating a music generation method according to some embodiments of this disclosure. Figure 1As shown, the music generation method used in electronic devices includes the following steps.

[0035] In step S11, text information describing the target music is obtained.

[0036] In this embodiment of the disclosure, the text information may be, for example, style, related intention, or applicable scenario.

[0037] In step S12, the text information is input into the target first model to encode the text information, obtain the first text sequence, and predict the latent vector sequence that identifies the music based on the first text sequence.

[0038] In this embodiment of the disclosure, by encoding the text information, the model can understand the user needs contained in the text information. The first text information can be encoded using a pre-trained t5 encoder to obtain a first text sequence. Since the latent vector sequence output by the first model is used to identify music, the latent vector sequence identifying music can be decoded into music with good structure. The latent vector identifying music is latent∈[latent_dim, len], where dim represents the dimension and len represents the music length.

[0039] In step S13, the text information and the latent vector sequence are input into the second model to encode the text information and obtain the second text sequence. Based on the second text sequence and the latent vector sequence identifying music, the latent vector sequence identifying noise is predicted.

[0040] The noise here refers to the noise included in the implicit vector sequence that identifies the music.

[0041] In this embodiment, the latent vector sequence identifying the music, based on the second text sequence, can decode the musical effect, thus the second text sequence has higher encoding accuracy than the first text sequence. The second text sequence can be obtained by encoding with the same encoder that has higher accuracy than the encoder used to obtain the first text sequence. The second text sequence is used as a prefix of the latent vector sequence identifying the music and as a control condition for a cross-attention mechanism or an adaptive normalization mechanism (adaLN) to constrain the second model, enabling it to predict the noise present in the music described in the first text information and thereby remove that noise.

[0042] In step S14, the latent vector sequence of the identified music is denoised based on the latent vector sequence of the identified noise to obtain the denoised latent vector sequence, and the target music is generated based on the denoised latent vector sequence.

[0043] In this embodiment of the disclosure, the latent vector sequence of the identified music can be denoised by subtracting the latent vector sequence of the identified noise from the latent vector sequence of the identified music in the latent vector space. By decoding the denoised latent vector sequence, high-quality target music can be obtained, and since the target music is obtained based on the latent vector sequence of the identified music, it has high structure.

[0044] The above embodiments divide the process of generating target music into two stages. The first stage generates a latent vector sequence of the identified music based on text information, which can be decoded into well-structured music. The second stage uses the same text information to predict the generated latent vector sequence of the identified music, identifying and eliminating noise present in the sequence. Compared to methods that directly generate music, this two-stage approach yields music with both high musical quality and structural integrity.

[0045] Understandably, in order to improve the structural accuracy of the music generated by the first model, it is necessary to enable the first model to learn the structural features of the music. The following embodiments of this disclosure further illustrate how the model of this disclosure learns the structural features of music.

[0046] Figure 2 This is a flowchart illustrating how a first model learns the structural features of music according to some embodiments of this disclosure. Figure 2 As shown, the first model is trained in the following way, enabling it to learn the structural features of music.

[0047] In step S21, music samples and text information describing the music samples are obtained.

[0048] The text information used to describe music samples can be annotated manually.

[0049] In step S22, the text information describing the music sample is encoded to obtain a third text sequence, and each frame in the music sample is encoded to obtain the codebook sequence of the music sample.

[0050] Each frame corresponds to K codebooks, and the length of the codebook sequence of the music sample is the number of music sample frames.

[0051] In this embodiment of the disclosure, the encoder that encodes the text information used to describe the music sample to obtain the third text sequence can be, for example, a pre-trained text encoder t5, and the obtained third text sequence is used to provide the key vector and value vector of the CrossAttention mechanism in the first model to be trained.

[0052] The codebook can be used to represent the structural features and composition of music. The codebook is a set of discretized vectors used by the first model to represent music. A vector sequence is a sequence composed of vectors from the codebook arranged in a certain order. The encoder C1 that encodes the music samples can be, for example, a Residual Vector Quantization Variational Autoencoder (RVQ-VAE). Other encoders can also be used to encode the music samples; this embodiment does not impose specific limitations. After encoding the music samples using encoder C1, K parallel sequences of length equal to the number of frames are obtained, i.e., a codebook sequence of shape [K, len]. This codebook sequence is used as the training target of the first model to be trained.

[0053] In step S23, the third text sequence is input into the first model to be trained to obtain the hidden vector sequence that identifies the music.

[0054] In step S24, the cross-entropy loss is determined by performing K normalization calculations on the latent vector sequence of the identified music and K codebooks, and the first model to be trained is adjusted.

[0055] In this embodiment of the disclosure, the codebook can be understood as a classification dictionary with a fixed number of categories, and the cross-entropy loss can accurately measure the difference between the prediction result and the training objective, thereby providing an effective direction for optimization.

[0056] Repeat the above process until the cross-entropy loss meets the first condition, and the first model is obtained.

[0057] Through the above embodiments, by converting music samples into codebooks and preserving the structure of the music samples for the first model to learn, and by using text information annotation, the model learns how to generate music samples based on text information. This enables the first model to fully learn how to generate structured music samples, thereby improving the user experience.

[0058] Understandably, in music generation scenarios, situations often arise where music of varying durations needs to be generated. To ensure the music generation model can produce music with high structural integrity, it's crucial to ensure that the music samples used during model training fall within a set duration range. Therefore, a first model needs to be determined based on the target duration. The following embodiments further illustrate the method for generating music of the target duration in this disclosure.

[0059] Figure 3 This is a flowchart illustrating a method for generating music of a target duration according to some embodiments of the present disclosure.

[0060] like Figure 3As shown, the method for generating music of the target duration includes the following steps:

[0061] In step S31, text information describing the target music is obtained.

[0062] The music samples include multiple samples, and different samples correspond to different duration ranges. The first model includes multiple models, and different first models are trained based on music samples with different duration ranges.

[0063] The first text information includes the target duration, and the first model that predicts the latent vector sequence of the music corresponds to the target duration.

[0064] In step S32, the text information is input into the target first model to encode the text information, obtain the first text sequence, and predict the latent vector sequence that identifies the music based on the first text sequence.

[0065] In this embodiment, each first model is trained using only music samples within a specific duration range to ensure that the music generated by the first model within that specific duration range has high structural integrity. After training, there is a one-to-one correspondence between the first model, the music samples, and their durations. When music of a target duration needs to be generated, the corresponding first model can be invoked based on the target duration in the text information to predict the latent vector sequence identifying the music.

[0066] Through the above embodiments, the first model is trained using music samples within a specific duration range, establishing a one-to-one correspondence between the music samples, the first model, and the duration. This allows the first model to be determined based on the target duration when generating music, thus generating a latent vector sequence of the target duration's identifier music. This method ensures that the first model can generate a variety of highly structured target music tracks of different durations.

[0067] In this embodiment, because the music samples are encoded during the training of the first model, the encoding operation causes a certain degree of data distortion. Therefore, directly decoding the latent vector sequence of the identified music generated by the first model into the target music will result in insufficient sound effects. To improve the musical effect of the music after decoding the latent vector sequence of the identified music, noise removal can be used. The following embodiments further illustrate the method of enabling the model to predict noise in the latent vector sequence of the identified music.

[0068] Figure 4 This is a flowchart illustrating a method for enabling a model to predict noise in a sequence of latent vectors identifying music, according to some embodiments of this disclosure. Figure 4 As shown, the second model is trained in the following manner:

[0069] In step S41, text information describing the music sample and a vector sequence identifying the music obtained based on the third text sequence are acquired.

[0070] In step S42, the text information describing the music sample is encoded to obtain a fourth text sequence, and noise is added to the latent vector sequence obtained based on the third text sequence to obtain a latent vector sequence that identifies the noisy music.

[0071] In this embodiment, the same text information is used to train the first model and the second model to ensure that the second model can accurately predict noise in the latent vector sequence generated by the first model. However, since the second model is designed to further improve the sound quality of the third latent vector sequence, it needs to use a higher-precision encoding method to encode the second text information. The fourth text sequence can be a fixed-length text sequence, such as 128 characters. This fourth text sequence is used as a prefix input to the model and fed into the second model to be trained.

[0072] One method for adding noise to the latent vector sequence obtained from the third text sequence is to encode the noise-added sound using a pre-trained encoder C2 to obtain a noise-added sequence. The encoding quality of the encoder encoding the noise needs to be higher than that of the encoder encoding the music samples. The encoder C2 can be, for example, a Variational Autoencoder (VAE), or other encoders; this embodiment does not impose specific limitations. To enable the second model to accurately predict agitated noise, noise is added to the latent vector sequence obtained from the third text sequence until the latent vector sequence identifying the noise-added music can be considered at the level of white noise.

[0073] In step S43, the fourth text sequence and the latent vector sequence identifying the noise-added music are input into the second model to be trained to obtain the predicted noise-added noise. The second model to be trained is adjusted based on the predicted noise-added noise until the predicted noise-added noise is consistent with the added noise-added noise, thus obtaining the second model.

[0074] After inputting the fourth text sequence and the latent vector sequence identifying the noise-added music into the second model to be trained, the latent vector sequence obtained through the third text sequence is used as the control condition for the cross-attention mechanism or the adaptive normalization mechanism to predict the noise-added music.

[0075] In this embodiment, the music generated by the arbitrary music generation model can also be denoised and optimized using a second model. By encoding the music generated by the arbitrary music generation model to obtain a latent vector sequence of the identified music, noise is added to this latent vector sequence. The descriptive information used to generate the latent vector sequence of the identified music generated by the arbitrary music generation model, along with the noisy latent vector sequence of the identified music, is input into the second model. The second model is then adjusted based on the mean squared error loss (MSE_loss) between the predicted noise and the added noise, thus achieving denoising and optimization of the music generated by the arbitrary music generation model.

[0076] Through the above embodiments, by adding noise to the latent vector sequence of the identified music generated by the first model, a latent vector sequence of the identified noise-added music is obtained, and the second model learns how to predict the noise-added method from the latent vector sequence of the identified noise-added music, thereby improving the noise reduction accuracy of the latent vector sequence generated by the first model, and thus improving the sound effect of the generated target music and enhancing the user experience.

[0077] In the embodiments of this disclosure, the noise in the latent vector sequence identifying the music may contain high-order noise. Therefore, a single prediction can only slightly improve the musical effect of the music. Multiple rounds of noise prediction can be used to further improve the musical effect of the target music. The following embodiments further illustrate the method of improving the musical effect of the target music through multiple noise predictions.

[0078] Figure 5 This is a flowchart illustrating a method for improving the musical effect of target music through multiple noise predictions, according to some embodiments of this disclosure. Figure 5 The method shown for improving the musical quality of target music through multiple noise predictions includes the following steps:

[0079] In step S51, the latent vector sequence of the identified music is denoised for the first time based on the latent vector sequence of the identified noise, to obtain the latent vector sequence after the first denoising.

[0080] In step S52, the latent vector sequence after the first denoising and the text information are input into the second model to denoise the latent vector sequence and obtain the latent vector sequence after the second denoising.

[0081] In step S53, the latent vector sequence after the second denoising is used as the denoised latent vector sequence.

[0082] The denoised latent vector sequence can be denoised multiple times to improve the sound effect, depending on the desired sound quality of the generated music.

[0083] Through the above embodiments, by predicting and denoising the noise in the latent vector sequence of the identified music, it is possible to eliminate the high-order noise that may exist in the latent vector sequence of the identified music generated by the first model, thereby further improving the musical effect of the generated target music and enabling the target music to have both high musical effect and structural integrity.

[0084] Based on the same concept, this disclosure also provides a music generation device 100.

[0085] It is understood that the music generation apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0086] Figure 6 This is a block diagram of a music generation apparatus 100 according to some embodiments of the present disclosure. (Refer to...) Figure 6 The device includes an acquisition module 101, a generation module 102, and an optimization module 103.

[0087] The acquisition module 101 is configured to acquire text information describing the target music.

[0088] The generation module 102 is configured to input text information into a target first model to encode the text information, obtain a first text sequence, and predict a latent vector sequence identifying music based on the first text sequence.

[0089] The optimization module 103 is configured to input text information and latent vector sequence into the second model to encode the text information to obtain a second text sequence, and predict the latent vector sequence of the identified noise based on the second text sequence and the latent vector sequence of the identified music, wherein the noise is the noise included in the latent vector sequence of the identified music; based on the latent vector sequence of the identified noise, the latent vector sequence of the identified music is denoised to obtain a denoised latent vector sequence, and the target music is generated based on the denoised latent vector sequence.

[0090] In some embodiments, the first model is trained as follows: music samples and text information describing the music samples are acquired; the text information describing the music samples is encoded to obtain a third text sequence, and each frame in the music samples is encoded to obtain a codebook sequence for the music samples, wherein each frame corresponds to K codebooks, and the length of the codebook sequence for the music samples is the number of frames in the music samples; the third text sequence is input into the first model to be trained to obtain a latent vector sequence identifying the music; based on the latent vector sequence identifying the music, the vector after K normalization calculations and the K codebooks are used to determine the cross-entropy loss, and the first model to be trained is adjusted; the above process is repeated until the cross-entropy loss satisfies the first condition, thus obtaining the first model.

[0091] In some embodiments, there are multiple music samples, and different samples correspond to different duration ranges. There are multiple first models, and different first models are trained based on music samples with different duration ranges. The text information describing the target music includes the target duration, and the target first model is trained based on music samples with the duration range corresponding to the target duration.

[0092] In some embodiments, the second model is trained as follows: obtaining text information describing music samples and a vector sequence identifying music based on a third text sequence; encoding the text information describing music samples to obtain a fourth text sequence, and adding noise to the latent vector sequence obtained based on the third text sequence to obtain a latent vector sequence identifying the noise-added music; inputting the fourth text sequence and the latent vector sequence identifying the noise-added music into the second model to be trained to obtain the predicted noise; adjusting the second model to be trained based on the predicted noise until the predicted noise matches the added noise, thus obtaining the second model.

[0093] In some embodiments, the optimization module 103 denoises the latent vector sequence of the identified music based on the latent vector sequence of the identified noise in the following manner to obtain a denoised latent vector sequence: the latent vector sequence of the identified music is denoised for the first time based on the latent vector sequence of the identified noise to obtain a latent vector sequence after the first denoising; the latent vector sequence after the first denoising and the text information are input into the second model to denoise the latent vector sequence after the second denoising to obtain a latent vector sequence after the second denoising; the latent vector sequence after the second denoising is used as the latent vector sequence after denoising.

[0094] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0095] Figure 7This is a block diagram illustrating an apparatus 200 for music generation according to some embodiments of the present disclosure. For example, apparatus 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0096] Reference Figure 7 The device 200 may include one or more of the following components: processing component 202, memory 204, power supply component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0097] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0098] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0099] Power supply component 206 provides power to various components of device 200. Power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 200.

[0100] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0101] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0102] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0103] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0104] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 216 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0105] In some embodiments of this disclosure, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0106] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by the processor 220 of the device 200 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0107] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0108] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of the music generation device, the music generation device is able to perform the methods described above.

[0109] Figure 8 This is a block diagram illustrating an apparatus 300 for music generation according to some embodiments disclosed in a publication. For example, apparatus 300 may be provided as a server. See also... Figure 8 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the methods described above.

[0110] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0111] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 332 including instructions, which can be executed by the processing component 322 of the device 300 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0112] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0113] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of the music generation device, the music generation device is able to perform the methods described above.

[0114] In some embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the music generation method involved in any of the above embodiments.

[0115] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this disclosure can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented in hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this disclosure.

[0116] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, a first component, part, region, layer, or section mentioned in the examples may also be referred to as a second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature.

[0117] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0118] In this description, “multiple” means at least two, referring to two or more, such as two, three, etc., unless otherwise explicitly specified. Other quantifiers are similar. The singular forms “a,” “the,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, unless otherwise specified or clearly indicated from the context, the articles “a” and “one” as used in this disclosure and the appended claims are generally understood to mean “one or more.”

[0119] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of two or more; "and / or" describes the association relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Similarly, "at least one of..." includes any one of the related listed items and any combination of two or more.

[0120] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0121] Furthermore, the term "exemplary" is used herein to indicate that it serves as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term "exemplary" is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to indicate an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to indicate any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then applying A or B satisfies the condition under any of the foregoing instances.

[0122] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if it is not structurally equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in this disclosure, such terms are intended to be inclusive in a manner similar to the term “including.”

[0123] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0124] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating music, characterized in that, include: Obtain the text information used to describe the target music; The text information is input into the target first model to encode the text information to obtain a first text sequence, and a latent vector sequence identifying music is predicted based on the first text sequence. The text information and the latent vector sequence are input into the second model to encode the text information to obtain the second text sequence. Based on the second text sequence and the latent vector sequence of the identified music, the latent vector sequence of the identified noise is predicted, wherein the noise is the noise included in the latent vector sequence of the identified music. Based on the latent vector sequence of the identified noise, the latent vector sequence of the identified music is denoised to obtain a denoised latent vector sequence, and the target music is generated based on the denoised latent vector sequence.

2. The music generation method according to claim 1, characterized in that, The first model was trained in the following manner: Obtain music samples and text information used to describe the music samples; The text information describing the music sample is encoded to obtain a third text sequence, and each frame in the music sample is encoded to obtain a codebook sequence of the music sample, wherein each frame corresponds to K codebooks, and the length of the codebook sequence of the music sample is the number of frames of the music sample. The third text sequence is input into the first model to be trained to obtain the hidden vector sequence that identifies the music; Based on the latent vector sequence of the identified music, the vector after K normalization calculations and the K codebooks are used to determine the cross-entropy loss and adjust the first model to be trained. Repeat the above process until the cross-entropy loss satisfies the first condition, and obtain the first model.

3. The music generation method according to claim 2, characterized in that, The music samples include multiple samples, and different samples correspond to different duration ranges. The first model includes multiple models, and different first models are trained based on music samples with different duration ranges. The text information describing the target music includes the target duration, and the first target model is trained based on music samples within the duration range corresponding to the target duration.

4. The music generation method according to claim 2, characterized in that, The second model was trained in the following manner: Obtain the text information describing the music sample and the vector sequence identifying the music obtained based on the third text sequence; The text information describing the music sample is encoded to obtain a fourth text sequence, and noise is added to the latent vector sequence obtained based on the third text sequence to obtain a latent vector sequence that identifies the noise-added music. The fourth text sequence and the latent vector sequence of the noise-added music are input into the second model to be trained to obtain the predicted noise. The second model to be trained is adjusted based on the predicted noise until the predicted noise matches the added noise, thus obtaining the second model.

5. The music generation method according to claim 1, characterized in that, The process of denoising the latent vector sequence of the identified music based on the identified noise to obtain a denoised latent vector sequence includes: The latent vector sequence of the identified music is first denoised based on the latent vector sequence of the identified noise to obtain the latent vector sequence after the first denoising. The latent vector sequence after the first denoising and the text information are input into the second model to denoise the latent vector sequence, resulting in the latent vector sequence after the second denoising. The second denoised latent vector sequence is used as the denoised latent vector sequence.

6. A music generation device, characterized in that, include: The acquisition module is configured to acquire text information describing the target music. The generation module is configured to input the text information into a target first model to encode the text information to obtain a first text sequence, and predict a latent vector sequence identifying music based on the first text sequence. An optimization module is configured to input the text information and the latent vector sequence into a second model to encode the text information to obtain a second text sequence, and to predict a latent vector sequence of the identified noise based on the second text sequence and the latent vector sequence of the identified music, wherein the noise is the noise included in the latent vector sequence of the identified music; Based on the latent vector sequence of the identified noise, the latent vector sequence of the identified music is denoised to obtain a denoised latent vector sequence, and the target music is generated based on the denoised latent vector sequence.

7. The music generation apparatus according to claim 6, characterized in that, The first model was trained in the following manner: Obtain music samples and text information used to describe the music samples; The text information describing the music sample is encoded to obtain a third text sequence, and each frame in the music sample is encoded to obtain a codebook sequence of the music sample, wherein each frame corresponds to K codebooks, and the length of the codebook sequence of the music sample is the number of frames of the music sample. The third text sequence is input into the first model to be trained to obtain the hidden vector sequence that identifies the music; Based on the latent vector sequence of the identified music, the vector after K normalization calculations and the K codebooks are used to determine the cross-entropy loss and adjust the first model to be trained. Repeat the above process until the cross-entropy loss satisfies the first condition, and obtain the first model.

8. The music generation apparatus according to claim 6 or 7, characterized in that, The music samples include multiple samples, and different samples correspond to different duration ranges. The first model includes multiple models, and different first models are trained based on music samples with different duration ranges. The text information describing the target music includes the target duration, and the first target model is trained based on music samples within the duration range corresponding to the target duration.

9. The music generation device according to claim 7, characterized in that, The second model was trained in the following manner: Obtain the text information describing the music sample and the vector sequence identifying the music obtained based on the third text sequence; The text information describing the music sample is encoded to obtain a fourth text sequence, and noise is added to the latent vector sequence obtained based on the third text sequence to obtain a latent vector sequence that identifies the noise-added music. The fourth text sequence and the latent vector sequence of the noise-added music are input into the second model to be trained to obtain the predicted noise. The second model to be trained is adjusted based on the predicted noise until the predicted noise matches the added noise, thus obtaining the second model.

10. The music generation apparatus according to claim 6, characterized in that, The optimization module denoises the latent vector sequence of the identified music based on the latent vector sequence of the identified noise in the following manner to obtain a denoised latent vector sequence: The latent vector sequence of the identified music is first denoised based on the latent vector sequence of the identified noise to obtain the latent vector sequence after the first denoising. The latent vector sequence after the first denoising and the text information are input into the second model to denoise the latent vector sequence, resulting in the latent vector sequence after the second denoising. The second denoised latent vector sequence is used as the denoised latent vector sequence.

11. A music generation device, characterized in that, include: processor; Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the music generation method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a music generation processor, enable music generation to perform the steps of any one of claims 1 to 5 of the music generation method.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the music generation method as described in any one of claims 1 to 5.

Images