Music generation device, music generation method, and program

Abstract

Provided is a musical composition generation device that comprises an encoder that converts audio data for first and second musical compositions to first and second feature vectors within a first feature space and a decoder that generates audio data on the basis of a third feature vector obtained by synthesizing the first and second feature vectors.

Description

Technical Field

[0001] The present invention relates to a music generation device ,easy a music generation method and a program.

Background Art

[0002] In a DJ performance, a crossfade from a preceding song to a subsequent song is often used. During the crossfade, since the preceding song and the subsequent song are mixed and played, it is necessary to make devices such as matching the respective beat positions, avoiding dissonant sounds, and avoiding competition of vocal sounds. However, technologies for easily performing a crossfade with a good listening feeling by selecting in advance songs with good compatibility are described in, for example, Patent Document 1 and Patent Document 2.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, even for songs with good compatibility, for example, when the number of sounds increases during a crossfade, it may sound messy, or even if the keys match, there may be a conflict in harmony, and there is still room for improvement in terms of listening feeling. Also, limiting the crossfade to between songs with good compatibility can also be a constraint in DJ performances.

[0005] Therefore, an object of the present invention is to provide a music generation device ,easy a music generation method and a program that can crossfade songs with a natural listening feeling by converting audio data into feature amounts.

Means for Solving the Problems

[0006] [1] A music generation device comprising an encoder that converts the audio data of the first and second music pieces into first and second feature vectors in a first feature space, and a decoder that generates audio data based on a third feature vector synthesized from the first and second feature vectors. [2] The music generation device according to [1], wherein the first and second feature vectors are synthesized at a ratio set according to a user operation. [3] The music generation device according to [2], wherein the user operation includes an operation of cross-fading the first music piece into the second music piece. [4] When the encoder and the decoder convert the audio data of a music piece into a feature vector in the first feature space by the encoder, generate audio data based on the feature vector in the first feature space by the decoder, and learn an identification model to identify the audio data of the music piece and the generated audio data, the generated audio data is learned not to be identified by the identification model as the audio data of the music piece. The music generation device according to any one of [1] to [3]. [5] The encoder converts the audio data of the first and second music pieces into the first and second feature vectors in the first feature space and the fourth and fifth feature vectors in a second feature space, and the decoder synthesizes the first and second feature vectors at a first ratio to obtain the third feature vector, and generates audio data based on the sixth feature vector synthesized from the fourth and fifth feature vectors at a second ratio. The music generation device according to [1]. [6] The music generation device according to [5], wherein the first ratio and the second ratio are set independently of each other. [7] The music generation device according to [5] or [6], wherein the encoder and decoder pitch-shift the audio data of a song and then convert it into a feature vector in the first feature space and a feature vector in the second feature space by the encoder, and generate audio data based on a vector obtained by applying a vector rotation in the opposite direction to the pitch shift to the feature vector in the first feature space and the feature vector in the second feature space by the decoder, and when an identification model is trained to distinguish between the audio data of the song and the generated audio data, the generated audio data is trained so that it is no longer distinguished as the audio data of the song by the identification model. [8] A music similarity determination device comprising an encoder that converts audio data of first and second music pieces into first and second feature vectors in a first feature space, determines the similarity between the first and second music pieces from the distance between the first and second feature vectors, wherein the encoder converts the audio data of the music pieces into feature vectors in the first feature space, the decoder generates audio data based on the feature vectors in the first feature space, and the device is trained to distinguish between the audio data of the music pieces and the generated audio data, such that the generated audio data is no longer distinguishable as the audio data of the music pieces by the identification model. [9] A method for generating music, comprising an encoder that converts audio data of a song into feature vectors in a first feature space, a decoder that generates audio data based on the feature vectors in the first feature space, and an identification model that is trained to distinguish between the audio data of the song and the generated audio data, the method comprising: a learning step of training the encoder and the decoder so that the generated audio data is no longer distinguishable as the audio data of the song by the identification model; and a music generation step of using the encoder to convert audio data of first and second songs into first and second feature vectors in the first feature space, and using the decoder to generate audio data based on a third feature vector obtained by synthesizing the first and second feature vectors.

[10] A program for causing a computer to function as an encoder that converts audio data of the first and second musical pieces into first and second feature vectors in a first feature space, and a decoder that generates audio data based on a third feature vector obtained by combining the first and second feature vectors. [Brief explanation of the drawing]

[0007] [Figure 1] This figure conceptually illustrates an embodiment of the present invention. [Figure 2] This figure shows an example of a system configuration in which the elements shown in Figure 1 are implemented. [Figure 3] This diagram conceptually illustrates the learning steps in the first embodiment of the present invention. [Figure 4] This diagram conceptually illustrates the music generation step in the first embodiment of the present invention. [Figure 5] This diagram conceptually illustrates the learning steps in a second embodiment of the present invention. [Figure 6] This diagram conceptually illustrates the music generation step in a second embodiment of the present invention. [Figure 7] This figure shows a specific example of the configuration in Figure 6. [Modes for carrying out the invention]

[0008] Figure 1 is a conceptual diagram illustrating an embodiment of the present invention. First, the learning step of the encoder / decoder 100 is performed using a group of songs (MCs). As will be described later, the encoder converts the audio data of the songs into feature vectors in the feature space, and the decoder generates audio data based on the feature vectors. For learning, a discrimination model 200 is used in addition to the encoder / decoder 100. For various songs included in the group of songs (MCs), the discrimination model 200 is trained to distinguish between the audio data of the original songs and the audio data generated by the encoder / decoder 100, and the encoder / decoder 100 is trained so that the generated audio data is no longer distinguishable from the audio data of the original songs by the discrimination model 200.

[0009] Next, a music generation step is performed using the trained encoder / decoder 100. In the music generation step, the trained encoder / decoder 100 is used to generate music MCx from a feature vector created by combining the feature vectors obtained by transforming the two music tracks MC1 and MC2. For example, when crossfading music MC1 into music MC2, playing music MCx during the crossfade prevents an increase in the number of notes and key and chord collisions, allowing the crossfade to be performed while maintaining the playback of musical audio data.

[0010] Figure 2 shows an example of a system configuration in which the elements shown in Figure 1 are implemented. The system 10 illustrated in Figure 2 includes a PC (Personal Computer) 11, a DJ controller 12, speakers 13, and a server 14. The PC 11 is a device that stores, processes, and plays back audio data of music, and is not limited to a PC; it may also be a terminal device such as a tablet or smartphone. The DJ controller 12 is connected to the PC 11 via a communication means such as USB (Universal Serial Bus), and acquires user input related to music playback through channel faders, crossfaders, performance pads, jog dials, and various knobs and buttons. The audio data is played back using the speakers 13 according to the control of the DJ controller 12. The server 14 is connected to the PC 10 via a network as needed and provides various functions.

[0011] The encoder / decoder 100 and identification model 200 described above with reference to Figure 1 are implemented in software, for example, by a computer operating according to a program stored in memory. The learning step of the encoder / decoder 100 and the music generation step using the learned encoder / decoder 100 may be performed on the same device or on different devices. More specifically, in the example of system 10 shown in Figure 2, both the learning step and the music generation step may be performed on PC 11. Alternatively, the learning step may be performed on server 14 and the music generation step on PC 11. In this case, the parameters of the encoder / decoder 100 obtained through learning are sent from server 14 to PC 11. Alternatively, both the learning step and the music generation step may be performed on server 14. In this case, audio data of music MC1, MC2, or music identification information is sent from PC 11 to server 14, and audio data of music MCx generated by the learned encoder / decoder 100 is sent from server 14 to PC 11.

[0012] Furthermore, the track MCx does not necessarily have to be generated in real time during playback of tracks MC1 and MC2; pre-generated audio data of the track MCx may be played during the crossfade playback of tracks MC1 and MC2. Therefore, the PC 11 does not necessarily have to be connected to the DJ controller 12 or speaker 13, not only during the execution of the encoder / decoder 100 learning step but also during the execution of the track generation step using the encoder / decoder 100. Similarly, the learning step and the track generation step may be performed on a server 14 that is not network-connected to the PC 11. In other examples, at least part of the learning step and the track generation step may be performed on DJ equipment such as a mixer or an all-in-one DJ system (digital audio player with communication and mixing functions) instead of, or together with, the PC 11.

[0013] Below, we will further describe two embodiments of the encoder / decoder 100 configuration shown in Figure 1.

[0014] (First Embodiment) Figure 3 is a diagram conceptually illustrating the learning steps in a first embodiment of the present invention. In the illustrated example, the audio data of the song MC is input to the encoder 110 in intervals of a predetermined length (0.5 seconds as an example, not limited to this). The encoder 110 converts the input audio data into a feature vector VC in the feature space SP. The decoder 120 generates the audio data of the song MCz based on the feature vector VC.

[0015] In the learning step, adversarial learning is performed by the encoder 110 and decoder 120, as well as the discrimination model 200. Specifically, the discrimination model 200 is trained to distinguish the audio data of the original song MC as genuine and other audio data as fake, and the encoder 110 and decoder 120 are trained so that the song MCz generated from the feature vector VC is distinguished as genuine by the discrimination model 200. In other embodiments, the discrimination model is not necessarily used; for example, the encoder 110 and decoder 120 may be trained to increase the similarity between the audio data of the song MC and the audio data of the song MCz, or learning using the discrimination model and learning using similarity may be performed simultaneously.

[0016] Through the learning steps described above, the encoder 110 and decoder 120 are trained so that, for various songs shown as the song group MCs in Figure 1, the generated song MCz audio data is no longer distinguishable from the original song MC audio data by the identification model 200. When this training converges, the song MCz audio data generated using the encoder 110 and decoder 120 becomes indistinguishable from the original song MC audibly. Furthermore, the decoder 120 becomes capable of converting any feature vector in the feature space SP into musical audio data.

[0017] Figure 4 is a conceptual diagram illustrating the music generation step in the first embodiment of the present invention. In the illustrated example, the audio data of music MC1 and MC2 are input to the encoder 110 in intervals of predetermined length, the same as during training. The encoder 110 converts each input audio data into feature vectors VC1 and VC2 in the feature space SP. The decoder 120 generates the audio data of music MCx based on a feature vector VCx obtained by combining feature vectors VC1 and VC2 in a predetermined ratio. As described above, the decoder 120 is trained through the training step to be able to convert any feature vector in the feature space SP into musical audio data. Therefore, the audio data of music MCx generated by the decoder 120 in the music generation step becomes musical audio data.

[0018] The feature vector VCx is a composite of feature vectors VC1 and VC2, using a ratio set according to user operations, such as crossfading song MC1 to song MC2. For example, if the composition of feature vector VCx is expressed as VCx = r * VC1 + (1 - r) * VC2 using a ratio r, then by gradually changing the ratio r from 1 to 0, the feature vector VCx transitions between feature vector VC1 and feature vector VC2 within the feature space SP. As described above, the encoder 110 and decoder 120 are trained through the learning step so that the generated audio data is not audibly distinguishable from the audio data of the original song. Therefore, if the ratio r is 1, the audio data generated based on feature vector VCx is not audibly distinguishable from the audio data of song MC1, and if the ratio r is 0, the audio data generated based on feature vector VCx is not audibly distinguishable from the audio data of song MC2. By gradually changing the ratio r from 1 to 0, the audio data of song MCx can be gradually changed from audio data that is indistinguishable from song MC1 to audio data that is indistinguishable from song MC2 while maintaining its musical quality.

[0019] When performing a crossfade, both tracks MC1 and MC2 are being played, so the audio data input to the encoder 110 is updated at predetermined intervals (0.5 seconds, as an example not limited to the above). Therefore, the feature vectors VC1 and VC2 in the feature space SP also change over time. However, if they are continuous sections of the same song, the change in the position of the feature vectors in the feature space SP is relatively gradual. Consequently, the position of the feature vector VCx, which is a composite of feature vectors VC1 and VC2, in the feature space SP does not change abruptly. Therefore, the audio data of track MCx can be gradually changed while maintaining its musical quality throughout the crossfade from start to finish.

[0020] According to the first embodiment of the present invention as described above, during the crossfade from song MC1 to song MC2, song MCx is played based on a feature vector VCx, which is a composite of feature vectors VC1 and VC2 obtained by feature transformation of the audio data of songs MC1 and MC2. Since songs MC1 and MC2 are not played simultaneously, deterioration of the auditory experience due to an increase in the number of sounds or chord collisions can be avoided. Song MCx changes continuously from a song that is indistinguishable from song MC1 to a song that is indistinguishable from song MC2, for example, like morphing in video, and is maintained in a musical state, so that the crossfade from song MC1 to song MC2 can be performed with a natural auditory experience.

[0021] (Second embodiment) Figure 5 is a conceptual diagram illustrating the learning steps in a second embodiment of the present invention. In the illustrated example, the audio data of the song MC is input to two encoders 140A and 140B, respectively, in intervals of a predetermined length and after processing by pitch shift 130. Here, pitch shift 130 is a process that raises or lowers the pitch of the song MC by a random number of semitones. Encoder 140A processes the pitch-shifted audio data of the song MC into the feature space SP A Internal Feature Vector VC A Converted to, encoder 140B uses feature space SP B Internal Feature Vector VCB is converted. The decoder 150 generates audio data of the music MCz based on the vector obtained by adding the feature vector rotation 160 to the feature vector VC A and the feature vector VC B . The feature vector rotation 160 is a vector rotation opposite to the pitch shift 130, and causes a pitch change in the opposite direction by the same amount as the pitch shift 130. Here, at the start of learning, the feature vector rotation 160 does not necessarily cause a pitch change, but by learning the feature vector rotation together with the encoder 140A, at the time of learning convergence, it is learned so that a pitch change occurs in the output audio according to the rotation amount of the feature vector rotation 160.

[0022] Also in this embodiment, in the learning step, adversarial learning is performed by the encoders 140A, 140B and the decoder 150, and the discrimination model 200 in the same manner as in the first embodiment. Specifically, the discrimination model 200 is learned to discriminate the original music MC before the processing by the pitch shift 130 as genuine and other audio data as fake, and the encoders 140A, 140B and the decoder 150 are learned so that the music MCz generated from VC A , VC B is discriminated as genuine by the discrimination model 200.

[0023] Through the learning step as described above, for various music shown as the music group MCs in FIG. 1, the encoders 140A, 140B and the decoder 150 are learned so that the audio data of the generated music MCz is not discriminated from the audio data of the original music MC by the discrimination model 200. When this learning converges, the audio data of the music MCz generated using the encoders 140A, 140B and the decoder 150 becomes indistinguishable from the original music MC in terms of auditory sensation. Also, the decoder 150 can convert any feature vector within the feature spaces SP A , SP B into musical audio data.

[0024] Here, the feature vector VC AWhile the pitch shift 130 is processed by a feature vector rotation 160 (the opposite of the pitch shift) before being input to the decoder 150, the feature vector VC B Such processing is not applied, and the pitch change due to the pitch shift of 130 is reflected. Therefore, the feature vector VC B If the audio data of the song MCz generated based on this is trained so that it is no longer distinguishable from the original song MC audio data in the discrimination model 200, then the encoder 140B will generate the feature vector VC. B The encoder 140A is trained to extract features from the song's MC that are less affected by pitch shift, i.e., features that represent rhythm. On the other hand, the encoder 140A is trained as described above by the encoder 140B to create a feature vector VC. A The system is trained to extract features from the song's MC that are affected by pitch shift, i.e., features that represent the sense of pitch.

[0025] Figure 6 is a conceptual diagram illustrating the music generation step in a second embodiment of the present invention. In the illustrated example, the audio data of music MC1 and MC2 are input to encoders 140A and 140B in sections of the same length as during training. Encoder 140A processes each input audio data into the feature space SP. A Internal feature vector VC1 A VC2 A The encoder 140B converts each input audio data into a feature space SP. B Internal feature vector VC1 B VC2 B Convert to the feature space SP. The decoder 150 uses the synthesis unit 170A to convert to the feature space SP. A Internal feature vector VC1 A VC2 A Feature vector VCx, which is a composite of these elements at a predetermined ratio. A , and the composite section 170B is a characteristic space SP B Internal feature vector VC1 B VC2 B Feature vector VCx, which is a composite of these elements at a predetermined ratio. B Based on this, the audio data for the song's MCx file is generated.

[0026] In this embodiment, the decoder 150 uses the feature vector VCx to generate the music. A VCx B For each of these, the characteristic vector VC1 of song MC1. A VC1 B and the feature vector VC2 of the song MC2 A VC2 B The composition ratios of these elements can be set independently of each other. For example, feature vector VCx A The synthesis of VCx, with ratio r A ,r B When expressed using equations (1) and (2), the ratio r A ,r B Any value between 0 and 1 can be set for this. As described above, encoder 140A is trained to extract features representing the pitch of the song, and encoder 140B is trained to extract features representing the rhythm of the song, so the ratio r A ,r B This setting allows you to individually adjust the influence of the original tracks MC1 and MC2 on the pitch and rhythm of the MCx track.

[0027] VCx A =r A *VC1 A +(1-r A )*VC2 A ...(1) VCx B =r B *VC1 B +(1-r B )*VC2 B ...(2)

[0028] Figure 7 shows a specific example of the configuration in Figure 6. In the illustrated example, the ratio r is on the composite section 170A side. A =1, ratio r on the composite part 170B side B =0 is set. In this case, the feature vector VCx is input to decoder 150. A This is the characteristic vector VC1 of song MC1. A Equal to the feature vector VCx BThis is the characteristic vector VC2 of the song MC2. B This is equal to . In this case, the decoder 150 generates audio data for song MCx that reflects the characteristics of song MC1 in terms of pitch and the characteristics of song MC2 in terms of rhythm. Ratio r A ,r B This value may be fixed at a predetermined value during the crossfade, as in the example above, or it may change in response to user interaction with the crossfade.

[0029] According to the second embodiment of the present invention described above, in addition to being able to crossfade from song MC1 to song MC2 with a natural sound by playing song MCx, similar to the first embodiment described above, it is possible to control whether the musical elements of song MCx, namely the sense of pitch and rhythm, are closer to song MC1 or song MC2, thereby enabling the playback of a song that is closer to the user's intention in terms of sound.

[0030] In the above embodiment, a music generation device using a trained encoder and decoder was described. However, as another embodiment, a music similarity determination device is also possible, which uses only the encoder from the similarly trained encoder and decoder to determine the similarity of music MC1 and MC2 from the distance between feature vectors VC1 and VC2 in the feature space SP. In this case as well, for example, music MC1 and MC2 whose feature vectors VC1 and VC2 are located close together in the feature space SP and are determined to have a high degree of similarity can be cross-faded, thereby creating a cross-fading effect that sounds natural. [Explanation of Symbols]

[0031] 10...System, 12...DJ controller, 13...Speaker, 14...Server, 100...Encoder / Decoder, 110...Encoder, 120...Decoder, 130...Pitch shift, 140A...Encoder, 140B...Encoder, 150...Decoder, 160...Feature vector rotation, 170A...Synthesizer, 170B...Synthesizer, 200...Identification model.

Claims

1. An encoder that converts the audio data of the first and second songs into first and second feature vectors in the first feature space and third and fourth feature vectors in the second feature space, A decoder that generates audio data based on a fifth feature vector obtained by combining the first and second feature vectors in a first ratio, and a sixth feature vector obtained by combining the third and fourth feature vectors in a second ratio. A music generation device equipped with the following features.

2. The music generation apparatus according to claim 1, wherein the first ratio and the second ratio are set independently of each other.

3. The music generation apparatus according to claim 1 or 2, wherein the encoder and decoder pitch-shift the audio data of the music and then convert it into a feature vector in the first feature space and a feature vector in the second feature space by the encoder, and generate audio data based on a vector obtained by applying a vector rotation in the opposite direction to the pitch shift to the feature vector in the first feature space and the feature vector in the second feature space by the decoder, and when an identification model is trained to distinguish between the audio data of the music and the generated audio data, the generated audio data is trained so that it is no longer distinguishable as the audio data of the music by the identification model.

4. The music generation apparatus according to claim 1 or claim 2, wherein the first ratio is set according to user operation.

5. The music generation apparatus according to claim 4, wherein the user operation includes an operation to crossfade the first music to the second music.

6. A method for generating music by computer, An encoder that converts the audio data of a song into a feature vector in a first feature space and a feature vector in a second feature space, and a decoder that generates audio data based on the feature vector in the first feature space and the feature vector in the second feature space, are trained by an identification model that is trained to distinguish between the audio data of the song and the generated audio data, so that the generated audio data is no longer distinguishable from the audio data of the song, A music generation step comprising: converting audio data of first and second music pieces into first and second feature vectors in the first feature space and third and fourth vectors in the second feature space using the encoder; and generating audio data based on a fifth feature vector obtained by combining the first and second feature vectors in a first ratio using the decoder and a sixth feature vector obtained by combining the third and fourth feature vectors in a second ratio. A method for generating music that includes this.

7. An encoder that converts the audio data of the first and second songs into first and second feature vectors in the first feature space and third and fourth feature vectors in the second feature space, A decoder that generates audio data based on a fifth feature vector obtained by combining the first and second feature vectors in a first ratio, and a sixth feature vector obtained by combining the third and fourth feature vectors in a second ratio. A program that makes a computer function.

Images