Music generation method, system and device and storage medium
The preprocessing and discretization of music data through the VQ-VAE network based on sparse converter is solved, and the problems of low music generation quality and poor stability in the prior art are achieved, and higher quality and stable music generation are achieved.
Patent Information
- Application Number
- CN202410998988.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-26
AI Technical Summary
When faced with complex and changing music styles, existing music generation methods cannot accurately obtain the discreteness and structural characteristics of music, resulting in low quality and poor stability of the generated music.
Using a VQ-VAE network built on a sparse transformer, a continuous potential representation is obtained by preprocessing the music data set, and after discretization processing, the input decoder generates music, and multiple music sequence samples are used for training to capture the discreteness of music. and structural features.
Improve the quality and stability of music generation, ensuring that the generated music can more accurately reflect the characteristics of complex music styles.
Smart Images

Figure CN119028299A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of music generation, and in particular relates to a music generation method, system, device and storage medium. Background Art
[0002] Music is an important art form that expresses people's thoughts, feelings and lives, and is an important crystallization of human production and life. Nowadays, with the rapid development of deep learning and artificial intelligence technology, music generation has attracted more and more attention. As an interdisciplinary research field, music generation involves the deconstruction, analysis and learning of music structure, style and expression from raw music data to achieve the goal of automatically generating new music. This process requires not only that the music generation model can capture the basic features of music such as time and pitch, but also that it needs to understand more complex music structures, such as the deconstruction and learning of chords, rhythm and melody.
[0003] The music generation methods in the prior art usually rely on rules and templates, and generate music by inputting music scores. When faced with complex and changeable music styles, this method often cannot accurately obtain the discrete and structural characteristics of music, resulting in low quality and poor stability in the generated music. Summary of the invention
[0004] In order to solve the technical problem existing in the background technology that the existing music generation method cannot accurately obtain the discrete and structural characteristics of music when facing complex and changeable music styles, resulting in low quality and poor stability in the generated music, the present invention provides a music generation method, system, device and storage medium.
[0005] In order to achieve the above object, the present invention adopts the following technical solution:
[0006] In a first aspect, the present invention provides a music generation method, which includes: obtaining a music data set, preprocessing the music data set to obtain a first music sequence; inputting the first music sequence into an encoder of a VQ-VAE network to obtain a continuous latent representation; discretizing the continuous latent representation through a codebook of the VQ-VAE network to obtain a first discrete latent representation; inputting the first discrete latent representation into a decoder of the VQ-VAE network, decoding to obtain a second music sequence; generating music based on the second music sequence; wherein the VQ-VAE network is constructed based on a sparse transformer and is trained with multiple samples of the first music sequence.
[0007] Preferably, the step of preprocessing the music data set in the present invention includes: loading the music data set, obtaining the compound word representation of the first music sequence; wherein the compound word representation of the first music sequence is:
[0008]
[0009] Among them, S cp is the first music sequence, N represents the sequence length of the first music sequence; cp i A collection of compound word representations of the first music sequence; middle: A sequence of "families" of musical elements; A sequence of "beats" for musical elements; for the sequence of the musical element “rhythm”; A sequence of musical elements called "chords"; A sequence of musical elements called "tones"; A sequence of "sustained states" of musical elements; is a sequence of the music element "speed", and i is the sequence number of the first music sequence.
[0010] Preferably, the expression formula of the continuous potential representation in the present invention is:
[0011]
[0012] Among them, z e is the continuous potential representation, Enc(·) is the encoder of the VQ-VAE network, and S cp is the first music sequence, M is the preset number of the continuous potential representations, is the continuous potential representation z e The t-th representation vector in , t=1,…,M.
[0013] Preferably, the step of discretizing the continuous latent representation in the present invention includes: inputting the continuous latent representation into the codebook of the VQ-VAE network, calculating the Euclidean distance with each codeword in the codebook of the VQ-VAE network based on each representation vector in the continuous latent representation; and taking the codeword corresponding to the minimum Euclidean distance as the corresponding discrete vector representation in the first discrete latent representation.
[0014] Preferably, the continuous potential representation in the present invention is discretized by a codebook vector preset in the codebook of the VQ-VAE network, and the discretization formula is:
[0015]
[0016] Among them, the codebook vector is e=[e1,…,e K ] T ∈R K×L, K is the number of codewords preset in the codebook of the VQ-VAE network, and the codewords are individual vectors in the codebook of the VQ-VAE network; L is the dimension of the codeword, and is the same as the dimension of the continuous potential representation; e j is the j-th codeword in the codebook of the VQ-VAE network; is the continuous latent representation z e The tth representation vector in ; is the first discrete potential representation z q The tth representation vector in .
[0017] Preferably, the method of inputting the first discrete potential representation into the decoder of the VQ-VAE network to obtain the second music sequence in the present invention is described as follows:
[0018] S cp ′=Dec(z q )
[0019] Where Dec(·) is the decoder of the VQ-VAE network, z q is the first discrete latent representation, S cp ' is the second music sequence output by the decoder of the VQ-VAE network.
[0020] Preferably, the music generation method described in the present invention also includes: establishing a priori model based on the sparse transformer; inputting the first discrete latent representation into the decoder of the VQ-VAE network, and the step of decoding to obtain the second music sequence specifically includes: inputting the first discrete latent representation into the priori model; generating a second discrete latent representation based on the prior model; inputting the second discrete latent representation into the decoder of the VQ-VAE network, and decoding to obtain the second music sequence.
[0021] In a second aspect, the present invention also provides a music generation system, which includes: an acquisition module for acquiring a music data set and preprocessing the music data set to obtain a first music sequence; an encoding module for encoding the first music sequence to obtain a continuous latent representation; a calculation module for discretizing the continuous latent representation to obtain a first discrete latent representation; a decoding module for decoding the first discrete latent representation to obtain a second music sequence; and a generation module for generating and outputting music based on the second music sequence.
[0022] In a third aspect, the present invention also provides a music generating device, comprising: a storage component and a processing component; instructions are stored in the storage component; the instructions are generated based on any one of the music generating methods described above; and the processing component calls the instructions in the storage component.
[0023] In a fourth aspect, the present invention further provides a storage medium having instructions stored thereon, wherein the instructions are generated based on any of the above-mentioned music generation methods.
[0024] In the music generation method provided by the present invention, a music data set is obtained, and the music data set is preprocessed to obtain a first music sequence; the first music sequence is input into the encoder of the VQ-VAE network to obtain a continuous potential representation; the obtained continuous potential representation is discretized through the codebook of the VQ-VAE network to obtain a first discrete potential representation; the first discrete potential representation is input into the decoder of the VQ-VAE network, and the second music sequence is obtained by decoding; music is generated based on the second music sequence; wherein the VQ-VAE network is constructed based on a sparse transformer and trained by multiple samples of the first music sequence. In the present invention, after the music data set is preprocessed into the first music sequence, the VQ-VAE network is trained by the first music sequence, so that the VQ-VAE network obtains the discrete and structural features in the first music sequence, thereby making the generated music higher in quality and more stable. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a schematic diagram of the music generation method described in the present invention;
[0026] Figure 2 This is a schematic diagram of the structure of the VQ-VAE network described in the present invention.
[0027] Figure 3 is a structural diagram of the sparse transformer described in the present invention;
[0028] Figure 4 is a structural diagram of the VQ-VAE network encoder described in the present invention;
[0029] Figure 5 It is a structural diagram of the VQ-VAE network decoder described in the present invention;
[0030] Figure 6 Schematic diagram of the music generation method with a priori model described in the present invention
[0031] Figure 7 is a loss value variation diagram of the VQ-VAE network provided by an embodiment of the present invention;
[0032] Figure 8 is a loss value variation diagram of the prior model provided by an embodiment of the present invention;
[0033] Fig. 9 is a graph showing changes in the accuracy of the prior model provided by an embodiment of the present invention;
[0034] Fig.10 It is a time domain diagram described in the present invention. DETAILED DESCRIPTION
[0035] The present invention provides a music generation method, system, device and storage medium for generating music according to an acquired music data set.
[0036] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0037] The VQ-VAE (Vector Quantized Variational Autoencoder) network is a deep learning model that combines variational autoencoders and vector quantization techniques. The model generally consists of three parts: encoder, codebook, and decoder. The encoder and codebook convert the input data into discrete potential representations, and the decoder reconstructs the output from these discrete potential representations. VQ-VAE uses a preset codebook, which allows it to maintain high performance and high generation quality when processing complex data.
[0038] Embodiment 1:
[0039] Reference Figures 1 to 5 The steps of the music generation method provided in the present invention include:
[0040] S101: Acquire a music data set, and preprocess the music data set to acquire a first music sequence.
[0041] The process of preprocessing the acquired music data set in the present invention includes: loading the music data set, and acquiring a first music sequence represented by a compound word. Specifically, the compound word representation of the first music sequence is:
[0042]
[0043] Among them, S cp is the first music sequence, N represents the sequence length of the first music sequence; cp i A collection of compound word representations of a first musical sequence; middle: A sequence of "families" of musical elements; A sequence of "beats" for musical elements; for the sequence of the musical element “rhythm”; A sequence of musical elements called "chords"; A sequence of musical elements called "tones"; A sequence of "sustained states" of musical elements; is a sequence of music element "speed", i is the serial number of the first music sequence; the symbol set ε = {f, b, t, c, p, d, v} is a symbol set corresponding to the music element mark.
[0044] S102: Input the first music sequence into the encoder of the VQ-VAE network to obtain a continuous latent representation.
[0045] The expression formula of the continuous potential representation in the present invention is:
[0046]
[0047] Among them, z e is the continuous potential representation, Enc(·) is the encoder of the VQ-VAE network, and S cp is the first music sequence input to the encoder of the VQ-VAE network, M is the number of preset continuous potential representations, is the continuous potential representation z e The t-th representation vector in , t=1,…,M.
[0048] S103: Discretize the continuous latent representation through the codebook of the VQ-VAE network to obtain a first discrete latent representation.
[0049] The step of discretizing the continuous latent representation in the present invention includes: inputting the continuous latent representation into the codebook of the VQ-VAE network, calculating the Euclidean distance between each representation vector in the continuous latent representation and each codeword in the codebook of the VQ-VAE network; taking the codeword corresponding to the minimum Euclidean distance as the corresponding discrete vector representation in the first discrete latent representation.
[0050] The continuous potential representation described in the present invention is discretized by the codebook vector preset in the codebook of the VQ-VAE network, and the discretization formula is:
[0051]
[0052] Among them, the codebook vector is e=[e1,…,e K ] T ∈R K×L , K is the number of codewords preset in the codebook of the VQ-VAE network, and a codeword is an individual vector in the codebook of the VQ-VAE network; L is the dimension of the codeword, which is the same as the dimension of the continuous potential representation; R is a set of real numbers, R K×L is a vector space; e j is the j-th codeword in the codebook of the VQ-VAE network; is the continuous latent representation z e The tth representation vector in ; is the first discrete latent representation z q The tth representation vector in .
[0053] S104: Input the first discrete latent representation into the decoder of the VQ-VAE network, and decode to obtain a second music sequence.
[0054] The method of inputting the first discrete potential representation into the decoder of the VQ-VAE network to obtain the second music sequence in the present invention is described as follows:
[0055] S cp ′=Dec(z q )
[0056] Where Dec(·) is the decoder of the VQ-VAE network, z q is the first discrete potential representation, S cp ' is the second music sequence output by the decoder of the VQ-VAE network.
[0057] S105: Generate music based on the second music sequence.
[0058] Before executing the music generation method, a VQ-VAE network should be constructed based on a sparse transformer (SparseTransformer), and the VQ-VAE network should be trained through samples of multiple first music sequences; the constructed VQ-VAE network should include an encoder, a decoder and a codebook, the encoder is constructed based on a sparse transformer encoding module, and the decoder is based on a sparse transformer decoder component.
[0059] Furthermore, in the process of training the VQ-VAE network, the GeLU activation function and the custom Adam optimizer with a learning rate of lr=1e-4 can be used to optimize the program network. The training set with a batch size of 32 is input for training. The number of epochs is set to 100, and different numbers of rounds of training are performed. At the same time, the loss value of the model under different numbers of rounds is recorded.
[0060] Furthermore, in the process of training the VQ-VAE network through multiple samples of the first music sequence, the loss function used includes three parts: reconstruction loss, which is used to measure the difference between the music sequence generated by the decoder and the original input music sequence; quantization loss, which is used to constrain the distance between the encoder output and the codeword in the codebook; commitment loss, which is used to force the encoder output to be close to the selected codeword to ensure that the encoder output is stable during the quantization process. The formula of its loss function is:
[0061]
[0062] Among them, L is the loss function for training the improved VQ-VAE network, p(x|z q ) is the discrete potential vector z after quantization q The probability of occurrence when the input sequence is x, z e is the continuous latent variable output by the encoder, e is the codebook vector, sg[·] is the stop gradient operator; β is the loss weight, and those skilled in the art can set the loss weight as needed, which is not specifically limited in this embodiment.
[0063] In this embodiment, a music data set is obtained, and the music data set is preprocessed to obtain a first music sequence; the first music sequence is input into the encoder of the VQ-VAE network to obtain a continuous potential representation; the obtained continuous potential representation is discretized through the codebook of the VQ-VAE network to obtain a first discrete potential representation; the first discrete potential representation is input into the decoder of the VQ-VAE network, and the second music sequence is obtained by decoding; and music is generated based on the second music sequence. After the music data set is preprocessed into the first music sequence, the VQ-VAE network is trained through the first music sequence, so that the VQ-VAE network obtains the discrete and structural features in the first music sequence, thereby making the generated music higher in quality and more stable.
[0064] Furthermore, the music generation method described in the present invention may also include visually converting the music generated based on the second music sequence to output a time domain graph, such as Fig.10 shown.
[0065] The calculation formula for converting music visualization into a time domain diagram is:
[0066] t=len(audio) / freq
[0067] Among them, t represents the duration of the music, len(audio) represents the number of music sampling points, and freq represents the sampling frequency of the music.
[0068] By converting the generated music into a time domain graph, the output music can be viewed and compared more intuitively by users.
[0069] Embodiment 2:
[0070] Reference Figure 6 The steps of the music generation method provided in the present invention include:
[0071] S201: Acquire a music data set, and preprocess the music data set to acquire a first music sequence.
[0072] The process of preprocessing the acquired music data set in the present invention includes: loading the music data set, and acquiring a first music sequence represented by a compound word. Specifically, the compound word representation of the first music sequence is:
[0073]
[0074] Among them, S cp is the first music sequence, N represents the sequence length of the first music sequence; cp i A collection of compound word representations of a first musical sequence; middle: A sequence of "families" of musical elements; A sequence of "beats" for musical elements; for the sequence of the musical element “rhythm”; A sequence of musical elements called "chords"; A sequence of musical elements called "tones"; A sequence of "sustained states" of musical elements; is a sequence of the music element "speed", and i is the sequence number of the first music sequence.
[0075] S202: Input the first music sequence into the encoder of the VQ-VAE network to obtain a continuous latent representation.
[0076] The expression formula of the continuous potential representation in the present invention is:
[0077]
[0078] Among them, z e is the continuous potential representation, Enc(·) is the encoder of the VQ-VAE network, and S cp is the first music sequence, M is the number of preset continuous potential representations, is the continuous potential representation z e The t-th representation vector in , t=1,…,M.
[0079] S203: Discretize the continuous latent representation through the codebook of the VQ-VAE network to obtain a first discrete latent representation.
[0080] The step of discretizing the continuous latent representation in the present invention includes: inputting the continuous latent representation into the codebook of the VQ-VAE network, calculating the Euclidean distance between each representation vector in the continuous latent representation and each codeword in the codebook of the VQ-VAE network; taking the codeword corresponding to the minimum Euclidean distance as the corresponding discrete vector representation in the first discrete latent representation.
[0081] The continuous potential representation described in the present invention is discretized by the codebook vector preset in the codebook of the VQ-VAE network, and the discretization formula is:
[0082]
[0083] Among them, the codebook vector is e=[e1,…,e K ] T ∈R K×L , K is the number of codewords preset in the codebook of the VQ-VAE network, and a codeword is an individual vector in the codebook of the VQ-VAE network; L is the dimension of the codeword, and is the same as the dimension of the continuous latent representation; e j is the jth codeword in the codebook of the VQ-VAE network; is the continuous latent representation z e The tth representation vector in ; is the first discrete latent representation z q The tth representation vector in .
[0084] S204: constructing a priori model based on the sparse transformer, and training the priori model to obtain a trained priori model.
[0085] In the process of training the prior model, the first discrete potential representation output by the trained VQ-VAE network is input into the prior model for training, so that the prior model learns the distribution of the first discrete potential representation.
[0086] Furthermore, in the process of training the prior model, a negative log likelihood function (NLL) can be used as a loss function. It should be noted that those skilled in the art can select a loss function in the prior model training process according to actual needs, and this embodiment only provides a more preferred implementation scheme.
[0087] S205: Input the first discrete latent representation into the trained prior model to obtain a second discrete latent representation.
[0088] In the process of music generation by the present music generation method, the first discrete potential representation is input into the trained prior model to obtain the second discrete potential representation. In the absence of a priori model, it is necessary to sample the first discrete potential representation from the codebook of the VQ-VAE network, and decode the first discrete potential representation through the decoder of the VQ-VAE network to generate the second music sequence.
[0089] S206: Input the second discrete latent representation into the decoder of the VQ-VAE network, and decode to obtain a second music sequence.
[0090] The method of inputting the second discrete potential representation into the decoder of the VQ-VAE network to obtain the second music sequence in the present invention is described as follows:
[0091] S cp ′=Dec(z q )
[0092] Where Dec(·) is the decoder of the VQ-VAE network, z q is the first discrete potential representation, S cp ' is the second music sequence output by the decoder of the VQ-VAE network.
[0093] S207: Generate music based on the second music sequence.
[0094] Reference Figure 7 and Figure 8 , the loss values of the VQ-VAE network and the prior model gradually decrease from high to low, and finally tend to be stable, indicating that the model reconstruction ability and codebook quantization effect are gradually improved.
[0095] Reference Fig. 9 , the accuracy of the prior model gradually increases and eventually stabilizes, indicating that the matching degree between the potential representation generated by the prior model and the target sequence gradually improves.
[0096] In this embodiment, after constructing and training the prior model, since the prior model has learned the distribution of the first discrete latent representation, in the application, the first discrete latent representation is input into the prior model to obtain the trained second discrete latent representation, thereby making the generated second music sequence and music have higher quality.
[0097] Embodiment three:
[0098] The present invention further provides a music generation system, which is formed based on the method of embodiment 1 or embodiment 2, and may include:
[0099] An acquisition module 301 is used to acquire a music data set and pre-process the music data set to acquire a first music sequence;
[0100] An encoding module 302, configured to encode the first music sequence to obtain a continuous latent representation;
[0101] A calculation module 303 is used to discretize the continuous potential representation to obtain a first discrete potential representation;
[0102] A decoding module 304, configured to decode the first discrete latent representation to obtain a second music sequence;
[0103] The generation module 305 is used to generate and output music based on the second music sequence.
[0104] Embodiment 4:
[0105] The present invention also provides a music generating device, which includes: a storage component and a processing component; the storage component stores instructions; the instructions are generated based on any one of the music generating methods described above; the processing component calls the instructions in the storage component to execute any one of the music generating methods described above.
[0106] Embodiment five:
[0107] The present invention also provides a storage medium, on which instructions are stored, and the instructions are generated based on any of the above-mentioned music generation methods.
[0108] Specifically, the storage medium described in the present invention may include random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk or CD-ROM. It should be noted that those skilled in the art may select the form and type of storage medium according to actual production and use requirements, and this is not further limited in this embodiment.
[0109] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices and storage media can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
Claims
1. A music generation method, characterized in that: The music generation method comprises: Acquire a music data set, and preprocess the music data set to acquire a first music sequence; Inputting the first music sequence into an encoder of the VQ-VAE network to obtain a continuous latent representation; Discretize the continuous latent representation through the codebook of the VQ-VAE network to obtain a first discrete latent representation; Inputting the first discrete latent representation into the decoder of the VQ-VAE network, and decoding to obtain a second music sequence; generating music based on the second music sequence; The VQ-VAE network is constructed based on a sparse transformer and is trained using samples of multiple first music sequences.
2. The music generation method according to claim 1, characterized in that: The step of preprocessing the music data set includes: Loading the music data set to obtain compound word representations of the first music sequence; The compound word of the first music sequence is represented as: Among them, S cp is the first music sequence, N represents the sequence length of the first music sequence; {cp i } is a collection of compound word representations of the first music sequence; middle: A sequence of "families" of musical elements; A sequence of musical elements called "beats"; A sequence of musical elements "rhythm"; is a sequence of musical elements "chords"; is a sequence of musical elements "tones"; A sequence of "persistent states" of musical elements; is a sequence of music elements "speed", and i is the serial number of the first music sequence.
3. The music generation method according to claim 1, characterized in that: The representation formula of the continuous potential representation is: Among them, z e is the continuous potential representation, Enc(·) is the encoder of the VQ-VAE network, and S cp is the first music sequence, M is the preset number of the continuous potential representations, is the continuous potential representation z e The t-th representation vector in , t=1,…,M.
4. The music generation method according to claim 1, characterized in that: The step of discretizing the continuous potential representation comprises: The continuous latent representation is input into the codebook of the VQ-VAE network, and the Euclidean distance is calculated based on each representation vector in the continuous latent representation and each codeword in the codebook of the VQ-VAE network; the codeword corresponding to the minimum Euclidean distance is taken as the corresponding discrete vector representation in the first discrete latent representation.
5. The music generation method according to claim 4, characterized in that: The continuous potential representation is discretized by the codebook vector preset in the codebook of the VQ-VAE network, and the discretization formula is: Among them, the codebook vector is e=[e1,…,e K ] T ∈R K×L , K is the number of codewords preset in the codebook of the VQ-VAE network, and the codewords are individual vectors in the codebook of the VQ-VAE network; L is the dimension of the codeword, and is the same as the dimension of the continuous potential representation; e j is the j-th codeword in the codebook of the VQ-VAE network; is the continuous latent representation z e The tth representation vector in ; is the first discrete potential representation z q is the tth representation vector in ; M is the number of the first discrete potential representations.
6. The music generation method according to claim 1, characterized in that: The method of inputting the first discrete latent representation into the decoder of the VQ-VAE network to obtain the second music sequence is described as: S cp ′=Dec(z q ) Where Dec(·) is the decoder of the VQ-VAE network, z q is the first discrete latent representation, S cp ' is the second music sequence output by the decoder of the VQ-VAE network.
7. The music generation method according to claim 1, characterized in that: The music generation method further comprises: establishing a priori model based on the sparse transformer; Inputting the first discrete potential representation into the decoder of the VQ-VAE network, and decoding to obtain the second music sequence specifically includes: inputting the first discrete latent representation into the prior model; generating a second discrete latent representation based on the prior model; The second discrete latent representation is input into the decoder of the VQ-VAE network, and the second music sequence is obtained by decoding.
8. A music generation system, characterized in that: The music generation system comprises: An acquisition module, used for acquiring a music data set and preprocessing the music data set to acquire a first music sequence; An encoding module, configured to encode the first music sequence to obtain a continuous latent representation; A computing module, configured to discretize the continuous latent representation to obtain a first discrete latent representation; A decoding module, configured to decode the first discrete latent representation to obtain a second music sequence; A generation module is used to generate and output music based on the second music sequence.
9. A music generating device, characterized in that: The music generating device comprises: a storage component and a processing component; The storage component stores instructions, which are generated based on the music generation method according to any one of claims 1 to 7; The processing component calls the instructions in the storage component.
10. A storage medium, characterized in that: The storage medium stores instructions, wherein the instructions are generated based on the music generation method according to any one of claims 1 to 7.
