System and method for automatically composing music using approaches of generative adversarial network and adversarial inverse reinforcement learning algorithm
Patent Information
- Authority / Receiving Office
- TW · TW
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-03-10
- Publication Date
- 2022-09-16
Smart Images
Figure TWG2TA000874154_001 
Figure TWG2TA000874154_002 
Figure TWG2TA000874154_003
Abstract
Description
[Technical Field]
[0001] This invention relates to the technical field of automatic music composition systems, and more particularly to an automatic music composition system and method using adversarial generative networks and adversarial inverse reinforcement learning. [Previous Technology]
[0002] Music is an extremely important part of human life. It serves not only as a way to relax in daily life, but also, according to ancient Roman texts, has therapeutic effects and can improve mood. A special report indicates that music has the following seven major benefits: increased learning efficiency, stress relief, pain relief, enhanced memory, improved sleep, increased athletic performance, and a happier mood.
[0003] However, in traditional composition methods, composers must spend years learning instrumental skills and music theory, as well as many days, to complete a piece. Therefore, to enable people to efficiently create unique compositions without being limited by music theory knowledge and instrumental background, many automatic composition systems have been proposed. One such automatic composition system utilizes supervised deep learning algorithms. However, these supervised deep learning algorithms excessively reuse the same melodies, resulting in music that is unpleasant to listen to.
[0004] As can be seen from the above description, it is necessary to improve and redesign the existing automatic music composition system so that it can produce more pleasing and human-friendly music. In view of this, the inventor of this case has made great efforts in research and creation, and finally completed the development of an automatic music composition system and method using adversarial generative networks and inverse reinforcement learning. [Summary of the Invention]
[0005] The main objective of this invention is to provide an automatic music composition system and method using generative adversarial networks and inverse reinforcement learning. The automatic music composition system is applied to an electronic device, enabling the device to generate a piece of music based on a plurality of reference musical pieces. The system includes: a music database, a music feature extraction unit, a first computation module, a second computation module, and a third computation module. The music database stores a plurality of reference musical pieces. Specifically, the music feature extraction unit performs feature extraction processing on each of the reference musical pieces to extract a plurality of musical features. Next, the first computation module uses a deep learning algorithm to perform a first operation on the musical features to obtain at least one musical probability feature and at least one pre-trained weight parameter. Furthermore, the second computation module uses a reinforcement learning algorithm to perform a second operation on the plurality of musical features to obtain at least one note reward function. Furthermore, the third operation module uses a deep reinforcement learning algorithm to perform an initialization setting using the pre-trained weight parameters, and performs a third operation on a complex array of music theory data stored in a music theory database, the at least one note reward function, and the at least one pre-trained weight parameter, thereby obtaining a plurality of polyphonic reference music data; the polyphonic reference music data is pleasing and popular music.
[0006] To achieve the main objective of the present invention, the inventors of this case provide an embodiment of an automatic music composition system using generative adversarial networks and inverse reinforcement learning, which is applied to an electronic device to generate a piece of music based on a plurality of reference music data; the automatic music composition system includes: a music database for storing the plurality of reference music data; a music feature extraction unit for performing a feature extraction process on each of the reference music data to extract a plurality of music features; a first operation module for performing a first operation on the plurality of music features using a deep reinforcement learning algorithm to obtain at least one music probability feature and at least one pre-trained weight parameter; and a second operation module for performing a second operation on the plurality of music features using a reinforcement learning algorithm to obtain at least one note reward function; A third operation module is used to perform an initialization setting using a deep reinforcement learning algorithm with the at least one pre-trained weight parameter, and to perform a third operation on the complex array of music theory data stored in a music theory database, the at least one note reward function and the at least one pre-trained weight parameter, thereby obtaining a plurality of complex music data.
[0007] Furthermore, in order to achieve the main objective of the present invention, the inventors of this case also provide an embodiment of the automatic composition method using adversarial generative networks and inverse reinforcement learning, which is applied to an electronic device and implemented by a processor of the electronic device, and includes the following steps: (1) providing a music feature extraction unit to perform feature extraction processing on a plurality of music data stored in a music database of the electronic device, thereby extracting a plurality of music features; (2) providing a first operation module to perform a first operation on the plurality of music features using a deep learning algorithm, thereby obtaining at least one music probability feature and at least one pre-trained weight parameter; (3) providing a second operation module to perform a second operation on the plurality of music features using a reinforcement learning algorithm, thereby obtaining at least one note reward function; (4) A third operation module is provided to initialize the at least one pre-trained weight parameter using a deep reinforcement learning algorithm, and to perform a third operation on the complex array of music theory data stored in the music theory database, the at least one note reward function and the at least one pre-trained weight parameter, thereby obtaining a complex set of music data.
Implementation Method
[0009] In order to more clearly describe the automatic music composition system and method using adversarial generative networks and inverse reinforcement learning proposed in this invention, the preferred embodiments of this invention will be described in detail below with reference to the figures.
[0010] Figure 1 shows a perspective view of an electronic device using an automatic composition system employing adversarial generative networks and inverse reinforcement learning according to the present invention. Figure 2 shows a functional block diagram of the automatic composition system employing adversarial generative networks and inverse reinforcement learning according to the present invention. As shown in Figure 1, the automatic composition system 1 employing adversarial generative networks and inverse reinforcement learning according to the present invention is applied to an electronic device 2. In one embodiment, the automatic composition system 1 of the present invention is installed in an operating system (OS) of the electronic device 2, enabling a processor of the electronic device 2 to generate a piece of music based on a plurality of reference music data by executing the automatic composition system 1 of the present invention. As shown in Figure 2, the automatic composition system 1 of the present invention mainly includes: a music database 11, a music feature extraction unit 12, a first calculation module 13, a second calculation module 14, a third calculation module 15, and a music theory database 16. The music database is used to store multiple pieces of music data, and the music theory database 16 is used to store multiple pieces of music theory data. The music feature extraction unit 12 performs a feature extraction process on each of the reference music data to extract multiple music features. More specifically, the music features are recorded as the pitch of the music in MIDI (Musical Instrument Digital Interface) values. Figure 3 shows a schematic diagram of recording the pitch of the music. As shown in Figure 3, in the matrix on the right, the horizontal columns represent the chords played in each beat, and the vertical rows are the pitches recorded in MIDI values; Figure 3 shows a chord for 8 beats; from the matrix in Figure 3, it can be seen that the first note is played on two and a half beats. That is to say, the harmonic melody of the music can be recorded through this recording method. Next, the schematic matrix of the music feature extraction unit 12 is shown as follows (1):…………….(1)
[0011] In the above formula (1), N is the MIDI value of each pitch, T is the beat number, p records whether or not it is played, and a is the corresponding articulation. It is further explained that the first operation module 13 uses a deep reinforcement learning algorithm to perform a first operation on the plurality of music features, thereby obtaining at least one music probability feature and at least one pre-trained weights parameter. Figure 4 shows the functional block diagram of the first operation module. As shown in Figure 4, the first operation module 13 includes: a first operation unit 131, a second operation unit 132, a third operation unit 133 and a fourth operation unit 134. More specifically, the first operation unit 131 is used to convert each of the music features into a note vector feature; wherein the length of the note vector feature is 79, and it is composed of the current pitch vector feature, the current tone vector feature, the tone vector feature of the previous note, the tone vector feature of the next note, and the beat vector feature. Next, the second operation unit 132 performs a fourth operation along the time axis on the note vector features using a deep learning algorithm to obtain at least one time parameter. Furthermore, the third operation unit 133 performs a fifth operation along the note axis on the note vector features using a deep learning algorithm to obtain at least one note parameter. It is worth noting that the deep learning algorithm is a Long Short-Term Memory (LSTM) neural network algorithm. Specifically, the first operation module 13 performs the LSTM neural network algorithm along the time axis and note axis respectively through the second operation unit 132 and the third operation unit 133. In other words, the first operation module 13 of this invention employs a bi-axial architecture LSTM neural network algorithm.
[0012] As described above, the fourth operation unit 134 uses a deep learning algorithm (non-recursive linear algorithm) to perform a fifth operation on the time parameter and the note parameter, thereby obtaining at least one training weight parameter and the music probability feature composed of at least one note probability feature and at least one performance method probability feature. It should be noted that, to avoid overfitting of the long short-term memory neural network algorithm, the first operation module 13 of the present invention uses a dropout algorithm in the fourth and fifth operations.
[0013] As described above, the second operation module 14 is used to perform a second operation on the plurality of musical features using an enhancement learning algorithm, thereby obtaining at least one note reward function. More specifically, the enhancement learning algorithm is an adversarial inverse reinforcement learning (AIRL) algorithm. The adversarial inverse reinforcement learning algorithm is similar to the architecture of Generative Adversarial Guided Cost Learning (GAN-GCL); in other words, through the inverse reinforcement learning algorithm used by the second operation module 14, it obtains the maximum reward, thereby obtaining the note reward function. Herein, it simultaneously trains a generator and a discriminator. Herein, the discriminator can be derived using the following equations (2), (3) and (4). rt(s,a)=c* r mt(s,a)+(1-c)* r airl(s,a) ……………..(2) L(θ)=E[r(s,a)+ γ(maxa'Q(s',a';θ-)-Q(s,a;θ))2] …………….(3)………….…..(4)
[0014] In equations (2) and (3) above, c is a constant, s is the current state, a is the current action, s' is the next state, a' is the next action, r_mt is the music theory reward function, r_airl is the adversarial inverse reinforcement learning reward function, θ- is the weight of the target-Q network, and γ is the discount factor for future rewards. In equation (4) above, θ is the weight of the Q-network, and q(τ) is the generator density. Furthermore, the actual distribution p(τ) is represented by the Boltzmann distribution, and its reward function is the energy function. Continuing to explain the technology of the present invention, the third operation module 15 is used to perform an initialization setting using a deep reinforcement learning algorithm through the at least one pre-trained weight parameter, and to perform a third operation on the complex array of music theory data stored in a music theory database 16, the at least one note reward function, and the at least one pre-trained weight parameter, thereby obtaining a complex set of music data. The deep reinforcement learning algorithm used by the third operation module 15 in this embodiment is a Deep Q Learning network (DQN) algorithm combined with the aforementioned dual-axis long short-term memory neural network algorithm. More specifically, the inventors simulated and experimented with different values of the reward discount factor c, obtaining the following table (1): Value of c 0.25 0.5 0.75 Music Theory Rewards 32.55 34.27 34.15 AIRL Rewards 25.3 29.71 22.46 average value 28.93 31.99 28.31 Table (1)
[0015] Next, the inventors compared the polyphonic music data generated by the present invention with music data generated by different algorithms. The comparison is a difference comparison between the music data generated by the algorithm and existing human music (i.e., reference music data or master music data). The comparison results are shown in Table (2) below: Dual-axis LSTM Music Theory AIRL Music Theory + AIRL Ratio polyphonicity 0.108861 0.084123 0.040719 0.020159 Ratio steps in scale 0.301088 0.294629 0.092172 0.171911 Ratio empty bars 0.002749 0.000134 0.051174 0.020323 Ratio unique chords 0.540401 0.192479 0.026132 0.078249 Ratio chords repeated 0.157354 0.138524 0.085684 0.031908 Ratio tonality 0.022615 0.085325 0.058328 0.006202 Ratio chords in repeated motifs 0.275816 0.222622 0.270690 0.264218 Table (2)
[0016] As can be seen from Table (2) above, the difference comparison values of music theory and AIRL algorithm used in this embodiment are the lowest in three categories. It is worth noting that the lower the value, the more similar it is to music composed by humans. In other words, the smaller the difference value of the aforementioned difference comparison, the closer the music composed by the automatic composition system 1 of this invention is to music composed by humans. Furthermore, the inventors conducted a user preference survey on different algorithm architectures and human music, as shown in Table (3) below: Dual-axis LSTM Music Theory AIRL Music Theory + AIRL Human composition Total times preferred 20 66 30 69 75 Percentage preferred 19% 63% 29% 66% 72% Standard Deviation 4.02 4.91 4.62 4.82 4.57 Table (3)
[0017] As can be seen from Table (3) above, the degree of liking obtained by the music theory and AIRL algorithm architecture used in this embodiment is only lower than that of human-composed music, and the difference is small. From the objective analysis of Table (2) above and the subjective analysis of Table (3), it can be concluded that the music generated by the automatic composition system of the adversarial generative network and inverse reinforcement learning method of this invention is closest to human-composed music. In other words, the music generated by this invention has the advantages of being pleasant to listen to and being liked by humans.
[0018] Thus, the above has fully described an automatic composition system using adversarial generative networks and inverse reinforcement learning according to the present invention. Next, the following will continue to describe an automatic composition method using adversarial generative networks and inverse reinforcement learning according to the present invention. Referring again to Figures 1 and 2, and simultaneously to Figures 5 and 6, a first flowchart and a second flowchart of the automatic composition method using adversarial generative networks and inverse reinforcement learning according to the present invention are shown. The automatic composition method using adversarial generative networks and inverse reinforcement learning according to the present invention is applied to an electronic device 2, enabling the processor of the electronic device 2 to generate a piece of music based on a plurality of reference music data by executing the automatic composition method 1 of the present invention. As shown in Figures 5 and 6, the automatic composition method of the present invention includes multiple execution steps. First, in step S1, the music feature extraction unit 12 performs feature extraction processing on the plurality of reference music data stored in the music database 11, thereby extracting a plurality of music features.
[0019] As shown in Figures 5 and 6, the method flow then proceeds to step S2: the first operation module uses a deep learning algorithm to perform a first operation on the plurality of musical features, thereby obtaining at least one musical probability feature and at least one pre-trained weight parameter. It is worth noting that the method flow then proceeds to step S3: the second operation module (AIRL) uses a reinforcement learning algorithm to perform a second operation on the plurality of musical features, thereby obtaining at least one note reward function. Further, in step S4, the third operation module uses a deep reinforcement learning algorithm to initialize the at least one pre-trained weight parameter, and performs a third operation on the plurality of music theory data stored in the music theory database, the at least one note reward function, and the at least one pre-trained weight parameter, thereby obtaining a plurality of polyphonic reference musical data.
[0020] In more detail, step S2 includes the following steps: In step S21, a first operation unit 131 of the first operation module 13 converts each of the musical features into a note vector feature. Next, step S22 is executed, where a second operation unit 132 of the first operation module 13 performs a fourth operation along the time axis on the note vector features using a deep learning algorithm, thereby obtaining at least one time parameter. As shown in Figure 5, the method flow then executes step S23, where a third operation unit 133 of the first operation module 13 performs a fifth operation along the note axis on the note vector features using a deep learning algorithm, thereby obtaining at least one note parameter. Finally, a fourth operation unit 134 of the first operation module 13 performs a fifth operation on the time parameter and the note parameter using a deep learning algorithm (non-recursive linear algorithm), thereby obtaining at least one training weight parameter and the musical probability feature composed of at least one note probability feature and at least one performance method probability feature.
[0021] Thus, the above has fully and clearly described an automatic music composition system and method using adversarial generative networks and inverse reinforcement learning; and, as can be seen from the above, the present invention has the following advantages:
[0022] (1) The present invention mainly comprises a music database 11, a music feature extraction unit 12, a first operation module 13, a second operation module 14, and a third operation module 15, forming the automatic composition system 1 of the present invention using adversarial generative networks and inverse reinforcement learning. In particular, the music feature extraction unit 12 performs a feature extraction process on each of the reference music data to extract a plurality of music features. The music features are recorded in a matrix to represent the harmony of the music. In addition, the second operation module 14 uses an inverse reinforcement learning algorithm to perform a second operation on the plurality of music features to obtain at least one note reward function. The third operation module 15 uses a deep reinforcement learning algorithm to perform an initialization setting by at least one pre-trained weight parameter, and performs a third operation on the plurality of music theory data, the note reward function value, and the pre-trained weight parameter to obtain a plurality of polyphonic reference music data. Based on the above objective and subjective analysis, it can be concluded that the music produced by the automatic composition system 1 of this invention, which uses adversarial generative networks and inverse reinforcement learning, is closest to human-composed music. Furthermore, the music produced by the system of this invention is not only melodious but also more popular with humans.
[0023] It must be emphasized that the above detailed description is a specific description of feasible embodiments of the present invention. However, the embodiments are not intended to limit the patent scope of the present invention. All equivalent implementations or modifications that do not depart from the spirit of the present invention should be included in the patent scope of this case. [Simplified Explanation of the Diagram]
[0008] Figure 1 shows a perspective view of an electronic device using an automatic composition system employing adversarial generative networks and inverse reinforcement learning according to the present invention; Figure 2 shows a functional block diagram of the automatic composition system employing adversarial generative networks and inverse reinforcement learning according to the present invention; Figure 3 shows a schematic diagram of recording musical pitch; Figure 4 shows a functional block diagram of the first computation module; Figure 5 shows a first flowchart of the automatic composition method employing adversarial generative networks and inverse reinforcement learning according to the present invention; and Figure 6 shows a second flowchart of the automatic composition method employing adversarial generative networks and inverse reinforcement learning according to the present invention.
Claims
1. An automatic music composition system, applied in an electronic device, enabling the electronic device to generate a piece of music based on a plurality of reference musical data; the automatic music composition system comprising: A music database for storing a plurality of reference music pieces; a music feature extraction unit for performing feature extraction processing on each of the reference music pieces to extract a plurality of music features; a first operation module for performing a first operation on the plurality of music features using a deep reinforcement learning algorithm to obtain at least one music probability feature and at least one pre-trained weight parameter; a second operation module for performing a second operation on the plurality of music features using a reinforcement learning algorithm to obtain at least one note reward function; and a third operation module for performing an initialization setting using a deep reinforcement learning algorithm with the at least one pre-trained weight parameter, and performing a third operation on a plurality of music theory data stored in a music theory database, the at least one note reward function, and the at least one pre-trained weight parameter to obtain a plurality of complex music pieces.
2. The automatic music composition system as described in claim 1, wherein, The first computation module includes: a first computation unit for converting each of the musical features into a note vector feature; a second computation unit for performing a fourth operation along the time axis on the note vector features using a deep learning algorithm to obtain at least one time parameter; a third computation unit for performing a fifth operation along the note axis on the note vector features using a deep learning algorithm to obtain at least one note parameter; and a fourth computation unit for performing a fifth operation on the time parameter and the note parameter using a non-recursive deep learning algorithm to obtain at least one training weight parameter and the musical probability feature composed of at least one note probability feature and at least one performance method probability feature.
3. The automatic music composition system as described in claim 2, wherein, The first computation module uses a discard algorithm in the fourth and fifth operations to prevent the deep learning algorithm from overfitting.
4. The automatic music composition system as described in claim 2, wherein the deep learning algorithm is a long short-term memory neural network algorithm.
5. The automatic music composition system as described in claim 1, wherein the deep reinforcement learning algorithm is a deep Q-network learning algorithm.
6. The automatic music composition system as described in claim 4, wherein the deep reinforcement learning algorithm is a deep Q-network learning algorithm combined with the long short-term memory neural network algorithm, and the long short-term memory neural network algorithm is a dual-axis long short-term memory neural network algorithm.
7. The automatic music composition system as described in claim 1, wherein the reinforcement learning algorithm is an adversarial inverse reinforcement learning algorithm.
8. The automatic music composition system as described in claim 2, wherein, The length of the note vector feature is 79, and it is composed of the current pitch vector feature, the current tone vector feature, the tone vector feature of the previous note, the tone vector feature of the next note, and the beat vector feature.
9. The automatic music composition system as described in claim 1, wherein, The multiple note state features are based on MIDI values to store the pitch of each note.
10. An automatic composition method, applied in an electronic device and implemented by a processor of the electronic device, includes the following steps: (1) providing a music feature extraction unit to perform feature extraction processing on a plurality of music data stored in a music database of the electronic device, thereby extracting a plurality of music features; (2) providing a first operation module to perform a first operation on the plurality of music features using a deep learning algorithm, thereby obtaining at least one music probability feature and at least one pre-trained weight parameter; (3) providing a second operation module to perform a second operation on the plurality of music features using a reinforcement learning algorithm, thereby obtaining at least one note reward function; (4) providing a third operation module to initialize the at least one pre-trained weight parameter using a deep reinforcement learning algorithm, and to perform a third operation on a plurality of music theory data stored in the music theory database, the at least one note reward function and the at least one pre-trained weight parameter, thereby obtaining a plurality of complex music data.
11. The automatic composition method as described in claim 10, wherein, The first operation module includes a first operation unit, a second operation unit, a third operation unit, and a fourth operation unit, and step (2) further includes the following steps: (21) The first operation unit converts each of the musical features into a note vector feature; (22) The second operation unit uses a deep learning algorithm to perform a fourth operation along the time axis on the note vector feature, thereby obtaining at least one time parameter; (23) The third operation unit uses a deep learning algorithm to perform a fifth operation along the note axis on the note vector feature, thereby obtaining at least one note parameter; (24) The fourth operation unit uses a non-recursive deep learning algorithm to perform a fifth operation on the time parameter and the note parameter, thereby obtaining at least one training weight parameter and the musical probability feature composed of at least one note probability feature and at least one performance method probability feature.
12. The automatic composition method as described in claim 11, wherein, When performing the first operation, the first operation module uses a dropout algorithm to prevent the result of the deep learning algorithm from overfitting.
13. The automatic composition method as described in claim 10, wherein the deep learning algorithm is a long short-term memory neural network.
14. The automatic composition method as described in claim 10, wherein the deep reinforcement learning algorithm is a deep Q learning network (DQN) algorithm.
15. The automatic composition method as described in claim 14, wherein the deep reinforcement learning algorithm is a deep Q learning network (DQN) algorithm combined with the long short-term memory neural network algorithm, and the long short-term memory neural network algorithm is a dual-axis long short-term memory neural network algorithm.
16. The automatic composition method as described in claim 10, wherein the reinforcement learning algorithm is an inverse reinforcement learning algorithm.
17. The automatic composition method as described in claim 11, wherein, The length of the note vector feature is 79, and it consists of the current pitch vector feature, the current tone vector feature, the tone vector feature of the previous note, the tone vector feature of the next note, and the beat vector feature.
18. The automatic composition method as described in claim 10, wherein, The multiple note state features are based on MIDI values to store the pitch of each note.