Light-weight music generation method based on Mama

Abstract

The invention discloses a Mama-based lightweight music generation method, which comprises the following steps of: flattening a 2D Mel frequency spectrum through a frequency spectrum scanning strategy according to the unique property of music, constructing a token sequence, carrying out feature fusion on the token sequence, a randomly generated time step length and a text condition which is given by a user and is used for controlling a music style, and carrying out remodeling, so as to obtain the Mama-based lightweight music. Finally predicted noise is obtained, target noise is constructed by using the noise, and positive target noise and negative target noise are obtained respectively; meanwhile, aiming at realizing continuous control of the music style, a low-rank adapter module is constructed, and predicted positive noise and negative noise are obtained through the low-rank adapter module according to text condition information which is given by a user and is used for controlling the music style; and respectively calculating positive loss and negative loss, and updating the low-rank adapter module by minimizing the sum of the positive loss and the negative loss.

Description

Technical Field

[0001] The present invention relates to the technical field of audio generation, and in particular to a lightweight music generation method based on Mamba. Background Art

[0002] 1. Text-to-audio generation

[0003] In the field of text-to-audio (TTA) generation, a variety of methods have been developed, each with its own unique implementation and trade-offs.

[0004] Vector Quantized Autoencoders (VQ-VAEs), such as the models used in Jukebox and VITS, facilitate efficient generation and capture complex musical structures by compressing audio into a discrete latent space. However, these methods can face challenges such as codebook collapse, which can weaken model capacity and generation quality.

[0005] Generative Adversarial Networks (GANs), such as SEGAN and MelGAN, are known for their fast and efficient generation capabilities, enabling high-quality audio generation with low latency. However, GANs are prone to mode collapse and training instability, resulting in insufficient output diversity.

[0006] Transformer-based models, such as FastSpeech, SpeechT5, and AudioLM, leverage self-attention mechanisms to model long-range dependencies in audio, enabling high-quality and context-sensitive generation. However, these models typically have high data requirements and high computational overhead, limiting their applicability in resource-constrained environments.

[0007] Diffusion models, such as DiffWave and DiffSinger, generate audio from noise through an iterative denoising process, providing fine-grained control over the generation process while exhibiting strong robustness to diversity. However, the U-Net architecture, due to its convolutional nature, is inherently more sensitive to local features. While self-attention layers have been introduced to alleviate this issue, if the self-attention mechanism is not optimally designed, the model may still struggle to capture long-range dependencies in audio sequences.

[0008] The Diffusion Transformer (DiT) excels at capturing long-range dependencies and scaling efficiently, and has been used for TTA generation, offering a promising research direction for music generation. However, similar to other Transformer-based architectures, the DiT model suffers from high memory consumption and slow training speed when applied to long audio sequences due to the computational complexity of its self-attention mechanism.

[0009] 2. Music style transfer

[0010] Music style transfer is a technique in music generation that uses deep learning to transform one musical style into another. Numerous studies have explored this topic using various deep learning frameworks: TimbreTron leverages image style transfer to achieve timbre transfer; Groove2Groove uses an encoder-decoder architecture for one-shot style transfer; Bonnici et al. combined a variational autoencoder (VAE) with a generative adversarial network to transfer timbre, applying it to vocals and instruments; and Pop2Piano employs a Transformer architecture to achieve style transfer from pop music to piano adaptations.

[0011] However, these methods usually require a large amount of training data and usually only support a single style transfer, making it difficult to generate high-quality and natural music between multiple styles. In order to achieve diversified music style transfer, Li et al. proposed a time-varying text inversion module based on the U-Net diffusion model. At the same time, ControlNet integrates conditional inputs (such as edge detection, Hough line detection, user graffiti, etc.) into the potential diffusion model through a lightweight adapter, thereby better controlling the image generation process and generating more conditional images. Uni-ControlNet accepts multiple pixel-level controls through a single adapter branch without specifying all control conditions at the same time, while ControlNet requires a separate adapter branch for each control condition. In addition, guided control based on inference time or optimization-based control allows users to adjust the generation direction in real time, thereby enhancing the controllability of the output.

[0012] However, current music style transfer methods can generate music in specific styles, but it is difficult to precisely control the strength of these styles. In this paper, we propose a lightweight plug-and-play fine-tuning module that acts like a "music knob" and allows users to fine-tune the strength of the style in the generated music.

[0013] Defects in the existing technology:

[0014] Existing text-to-audio generation models are computationally expensive to train due to the computational complexity of the self-attention mechanism, and they also lack sufficient fine-grained control for fine-tuning specific styles. Current approaches for text-to-audio can be categorized into four distinct approaches: those based on vector quantized autoencoders, those based on generative adversarial networks, those based on Transformers, and those based on diffusion models.

[0015] Vector quantized autoencoder-based methods compress audio into a discrete latent space, facilitating efficient generation and capturing complex musical structure. However, these methods can face challenges such as codebook collapse, which can compromise model capacity and generation quality. Generative adversarial network-based methods are known for their fast and efficient generation capabilities, enabling high-quality audio generation with low latency. However, GANs are prone to mode collapse and training instability, resulting in insufficient output diversity. Transformer-based models utilize self-attention mechanisms to model long-range dependencies in audio, enabling high-quality and contextual generation. However, these models are often data-intensive and computationally expensive, limiting their applicability in resource-constrained environments. Diffusion model-based methods generate audio from noise through an iterative denoising process, offering fine-grained control over the generation process while demonstrating strong robustness to diversity. However, the convolutional nature of the U-Net architecture makes it inherently more sensitive to local features. While self-attention layers are introduced to mitigate this issue, if the self-attention mechanism is not optimally designed, these models may still struggle to capture long-range dependencies in audio sequences. Summary of the Invention

[0016] The present invention mainly aims to overcome the deficiencies of the prior art and provides a lightweight music generation method based on Mamba.

[0017] The present invention is achieved through the following technical solutions:

[0018] A lightweight music generation method based on Mamba, comprising the following steps:

[0019] Step 1: Extract the Mel spectrum information of the original music;

[0020] Step 2: Perform latent space encoding on the music Mel spectrum information obtained in step 1;

[0021] Step 3: Combine the latent variable information obtained in step 2 with random Gaussian noise to perform 2D data modeling to obtain a sequence of N patches;

[0022] Step 4: Perform spectrum scanning on the sequence obtained in step 3 and expand each patch in the sequence;

[0023] Step 5: Linearly embed each patch obtained in step 4 into the token and add the position embedding to form a token sequence;

[0024] Step 6: Feature fusion is performed on the token sequence obtained in step 5, the randomly generated time step, and the text condition for controlling the music style given by the user;

[0025] Step 7: The token sequence after feature fusion obtained in step 6 is normalized by the linear normalization layer, then the dimension is adjusted by the linear layer, and then reshaped by the reshape layer to obtain the final predicted noise

[0026] Step 8: Use the noise predicted in step 7 To construct the target noise, we can get the positive target noise and negative target noise And build a low-rank adapter module, according to the text condition information of the user-given control music style, the predicted positive noise is obtained through the low-rank adapter module and negative noise The positive loss and negative loss are calculated separately, and the low-rank adapter module is updated by minimizing the sum of the positive loss and the negative loss.

[0027] In the above technical solution, in step 4, a spectrum scanning strategy is designed: through column scanning in sequence scanning, the input patch is expanded along the traversal path and converted into a frequency sequence. Furthermore, to better adapt to the spectrum scanning strategy, a bidirectional Mamba model is chosen to process the frequency sequence in both forward and reverse directions, thereby improving the processing of global context information in the audio.

[0028] In the above technical solution, in step 6, the time step is encoded into a vector, and the vector is sent to the adaptive layer normalization module for regression of size scale and displacement parameters, and then bidirectional Mamba processing is performed to obtain forward sequence features and reverse sequence features.

[0029] In the above technical solution, in step 6, for the text conditions given by the user to control the music style, the text conditions are first fed into the ClAP model to obtain cross-modal conditional information related to the music. Then, the obtained cross-modal conditional information and the token sequence processed by each layer of the bidirectional Mamba model are sent to the cross-attention module to capture the cross-modal relationship between text and music and obtain music representation features that integrate text semantic information.

[0030] In the above technical solution, in step 6, the token sequence obtained in step 5, the forward sequence features and reverse sequence features obtained according to the time step, and the music representation features obtained according to the text condition that integrate the text semantic information are fused to finally obtain the token sequence T after feature fusion. L .

[0031] In the above technical solution, for a set of text condition information (C t ,C + ,C - ), where Ct is the target concept used to specify the content of the generated music, C + It is a positive style concept, C - It is a negative style concept;

[0032] According to the text condition information of the control music style given by the user (C t ,C + ,C - ) in the positive style concept C + and negative style concept C - , the corresponding positive noise is generated by the low-rank adapter module and negative noise

[0033] Use the noise predicted in step 7 To construct the target noise, the target noise is shown as follows:

[0034]

[0035] The variable z represents the noise t Target Concept C t and the noise obtained by step 6 and step 7 at time step t; The variable z represents the noise t , Positive style concept C + and the noise obtained by step 6 and step 7 at time step t; The variable z represents the noise t , Negative style concept C - The noise obtained by step 6 and step 7 with time step t; scale is a control intensity. When scale is set to 1 and -1, positive target noise is generated respectively. and negative target noise

[0036] Finally, calculate and The L2 loss between them is then minimized.

[0037] The present invention has the following beneficial effects:

[0038] This paper achieves lightweight music generation in a latent diffusion model while retaining the scalability of the Transformer. Specifically, this paper proposes a Mamba-based lightweight music generation method. Based on the unique properties of music, a spectrum scanning strategy is used to flatten the 2D Mel-spectrogram and construct a token sequence. This sequence is then processed through forward Mamba and backward Mamba, fully focusing on the contextual information of the music. Furthermore, to achieve continuous control of musical style, a low-rank adapter module is constructed, allowing for continuous regulation of style intensity in the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the method flow of steps 1 to 7 of the present invention.

[0040] Figure 2 Schematic diagram of the method flow of step 8 of the present invention.

[0041] For ordinary technicians in this field, other relevant drawings can be obtained based on the above drawings without any creative work. DETAILED DESCRIPTION

[0042] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0043] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0044] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0045] The present invention provides a lightweight music generation method based on Mamba, comprising the following steps:

[0046] Step 1: Extract the Mel-spectrogram information of the original music. The original music here refers to the music to be modified. The ultimate goal of this invention is to modify the original music based on the text condition information of the music style control requirements provided by the user to generate the corresponding music that meets the user's required style.

[0047] Step 2: Perform latent space encoding on the music mel-spectrogram information obtained in step 1. That is, the high-dimensional data (here, the spectral information) is mapped / converted / compressed into a low-dimensional latent space to obtain a low-dimensional latent variable.

[0048] Step 3: Combine the latent variable information obtained in step 2 with random Gaussian noise ε to perform 2D data modeling to obtain a sequence of N patches.

[0049] In this processing step, unlike the traditional U-Net architecture that directly processes 2D data, the present invention first combines the latent variable with a random Gaussian noise ε (that is, destroys the latent variable into a completely noisy state) to obtain the noisy variable z t , and then a Patchify layer is used to model the 2D data in the 1D framework. The specific approach is: the variable z of the noise is t Divide it into N patches of equal size, thus forming a sequence of N patches, which is subsequently used as the input of the bidirectional Mamba model.

[0050] Step 4: Perform spectrum scanning on the sequence obtained in step 3 to expand each patch in the sequence.

[0051] Based on the characteristics of the Mel spectrum graph, the present invention designs a spectrum scanning strategy to obtain a single traversal path. Specifically, the spectrum scanning strategy is: when processing the Mel spectrum, the scanning strategy should take into account the specific characteristics of the Mel spectrum and music. Although the Mel spectrum is a two-dimensional representation, it can be regarded as a frequency sequence, where each patch represents the frequency and amplitude of the music at a specific time point. This frequency sequence can be compared to a note sequence. Through column scanning in the sequence scan, the input patch is expanded along the traversal path (from the upper left to the lower right) to convert it into a frequency sequence. Adding this strategy to the frequency axis can enhance the model's ability to capture harmonic and spectral relationships more effectively. Furthermore, in order to better adapt the spectrum scanning strategy, the bidirectional Mamba model (bidirectional state space model, a type of diffusion model) is selected to process the frequency sequence from both the forward and reverse directions, thereby improving the processing of global context information in the audio.

[0052] Step 5: Linearly embed each patch obtained in step 4 into a token of length D and add the position embedding Thus, a token sequence is constructed.

[0053] As shown in formula (5):

[0054] T0=[P 1 ;P 2 ;P 3 ;…;P N ]+E pos

[0055] Among them, P n A 1D vector representing the nth patch.

[0056] Step 6: The token sequence obtained in step 5 is fused with the randomly generated time step t and the text condition C given by the user to control the music style.

[0057] For the time step, it is encoded into a vector. Before applying the bidirectional Mamba, the vector is first sent to the adaptive layer normalization module (AdaLN) for regression of size scale and displacement parameters, and then bidirectional Mamba processing is performed to obtain forward sequence features and reverse sequence features;

[0058] For the user-given text condition C that controls the music style, the text condition is first fed into the ClAP model (Contrastive Language-Audio Pretraining) to obtain cross-modal conditional information related to the music. Then, the obtained cross-modal conditional information and the token sequence processed by each layer of the bidirectional Mamba model are sent to the cross-attention module to capture the cross-modal relationship between text and music and obtain music representation features that integrate text semantic information.

[0059] Finally, the token sequence obtained in step 5, the forward sequence features and reverse sequence features obtained according to the time step, and the music representation features obtained according to the text condition that integrate the text semantic information are fused to obtain the token sequence T after feature fusion. L .

[0060] Step 7: Token sequence T after fusion of features obtained in step 6 L , normalized by the linear normalization layer, then adjusted by the linear layer, and reshaped by the reshape layer to obtain the final predicted noise

[0061] f=Norm(T L )

[0062]

[0063] Among them, Norm represents the linear normalization layer, Reshape represents the reshaping layer, and Linear represents the linear layer.

[0064] Step 8: Use the noise predicted in step 7 To construct the target noise, we can get the positive target noise and negative target noise And build a low-rank adapter (LoRA) module, according to the text condition information of the user-given control music style, the predicted positive noise is obtained through the low-rank adapter (LoRA) module and negative noise The positive loss and negative loss are calculated separately, and the low-rank adapter (LoRA) module is updated by minimizing the sum of the positive loss and the negative loss.

[0065] The Low Rank Adapter (LoRA) module is used to fine-tune large-scale neural networks for specific tasks under limited resources by decomposing the low-rank weight update matrix △W into two smaller weight matrices and To achieve this. The low-rank decomposition matrix and Injected into the pre-trained query, key and value projection matrices of each cross-attention layer in the Mamba model. Through this low-rank adapter module, only update and At the same time, keeping the weights of the Mamba model frozen enables the Mamba model to learn discrete style concepts and significantly reduces the number of parameters that need to be trained.

[0066] For a set of text condition information (C t ,C + ,C - ), where C t is the target concept used to specify the content of the generated music, such as “A neutral piano music”, C + It is a positive style concept, C - It is a negative style concept, C + and C - Usually a pair of relative text, such as "happy" and "sad".

[0067] During the training process, the text condition information (C t ,C + ,C - ) in the positive style concept C + and negative style concept C - , the corresponding positive noise is generated by the Low Rank Adapter (LoRA) module and negative noise

[0068] Positive noise and negative noise is represented as follows:

[0069]

[0070] ε θ Represents the processing operation of the low-rank adapter module, which is based on the variable z of the noise t Target Concept C t , Positive style concept C + and time step t, generating positive noise The low-rank adapter module is based on the variable z of the noise t Target Concept C t , Negative style concept C - and time step t, generate negative noise

[0071] To use C t As the content to generate music, and control its style strength through the slider, we choose to use the noise predicted in step 7 To construct the target noise, the target noise is shown as follows:

[0072]

[0073] The variable z represents the noise t Target Concept C t and the noise obtained by step 6 and step 7 at time step t; The variable z represents the noise t , Positive style concept C + and the noise obtained by step 6 and step 7 at time step t; The variable z represents the noise t , Negative style concept C - The noise obtained by step 6 and step 7 with time step t; scale is a control intensity. During the training process, only the noise under the positive and negative style concepts needs to be predicted. and You can get it in C t The movable noise space when predicting noise under the condition This allows the positive strength of the generated music to increase as the scale increases during inference, and vice versa, when the scale is set to 1 and -1 to generate positive target noise respectively. and negative target noise

[0074] Finally, calculate and The L2 loss between , and then minimize the loss shown:

[0075]

[0076] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0077] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0078] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0079] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, Python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect via the Internet).

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0081] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A lightweight music generation method based on Mamba, characterized in that: The following steps are involved: Step 1: Extract the Mel spectrum information of the original music; Step 2: Perform latent space encoding on the music Mel spectrum information obtained in step 1; Step 3: Combine the latent variable information obtained in step 2 with random Gaussian noise to perform 2D data modeling to obtain a sequence of N patches; Step 4: Perform spectrum scanning on the sequence obtained in step 3 and expand each patch in the sequence; Step 5: Linearly embed each patch obtained in step 4 into the token and add the position embedding to form a token sequence; Step 6: Feature fusion is performed on the token sequence obtained in step 5, the randomly generated time step, and the text condition for controlling the music style given by the user; Step 7: The token sequence after feature fusion obtained in step 6 is normalized by the linear normalization layer, then the dimension is adjusted by the linear layer, and then reshaped by the reshape layer to obtain the final predicted noise Step 8: Use the noise predicted in step 7 To construct the target noise, we can get the positive target noise and negative target noise And build a low-rank adapter module, according to the text condition information of the user-given control music style, the predicted positive noise is obtained through the low-rank adapter module and negative noise The positive loss and negative loss are calculated separately, and the low-rank adapter module is updated by minimizing the sum of the positive loss and the negative loss.

2. The Mamba-based lightweight music generation method according to claim 1, characterized in that: In step 4, the input patch is expanded along the traversal path through the column scan in the sequence scan to convert it into a frequency sequence; the bidirectional Mamba model is selected to process the frequency sequence in both forward and reverse directions.

3. The Mamba-based lightweight music generation method according to claim 1, characterized in that: In step 6, the time step is encoded into a vector, and the vector is sent to the adaptive layer normalization module for regression of size scale and displacement parameters, and then bidirectional Mamba processing is performed to obtain forward sequence features and reverse sequence features.

4. The Mamba-based lightweight music generation method according to claim 3, characterized in that: In step 6, for the user-given text conditions for controlling the music style, the text conditions are first fed into the ClAP model to obtain cross-modal conditional information related to the music. Then, the obtained cross-modal conditional information and the token sequence processed by each layer of the bidirectional Mamba model are sent to the cross-attention module to capture the cross-modal relationship between text and music and obtain music representation features that integrate text semantic information.

5. The Mamba-based lightweight music generation method according to claim 4, characterized in that: In step 6, the token sequence obtained in step 5, the forward sequence features and reverse sequence features obtained according to the time step, and the music representation features obtained according to the text condition that integrate the text semantic information are fused to obtain the token sequence T after feature fusion. L .

6. The Mamba-based lightweight music generation method according to claim 1, characterized in that: For a set of text condition information (C t ,C + ,C - ), where C t is the target concept used to specify the content of the generated music, C + It is a positive style concept, C - It is a negative style concept; According to the text condition information of the control music style given by the user (C t ,C + ,C - ) in the positive style concept C + and negative style concept C - , the corresponding positive noise is generated by the low-rank adapter module and negative noise Use the noise predicted in step 7 To construct the target noise, the target noise is shown as follows: The variable z represents the noise t Target Concept C t and the noise obtained by step 6 and step 7 at time step t; The variable z represents the noise t , Positive style concept C + and the noise obtained by step 6 and step 7 at time step t; The variable z represents the noise t , Negative style concept C - The noise obtained by step 6 and step 7 with time step t; scale is a control intensity. When scale is set to 1 and -1, positive target noise is generated respectively. and negative target noise Finally, calculate and The L2 loss between them is then minimized.

Images