Information processing device, control method for information processing device, and program

Abstract

Extract only the melody from the performance data. [Solution] The information processing device 200 includes a processor 210 as a control unit that receives performance information input from an input unit such as an electric instrument's control unit for inputting musical performance information, as predetermined performance data such as MIDI data. The processor 210, as this control unit, extracts the melody from the performance data such as MIDI data based on velocity information (velocity value) in the performance data such as MIDI data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method of the information processing apparatus, and a program.

Background Art

[0002] Conventionally, performance data in the MIDI standard is known as performance information. For example, in Patent Document 1, pitch, volume, and timbre are specified as features from MIDI standard data, and an evaluation apparatus for evaluating the skill of singing or performance is described for a plurality of evaluation items such as "pitch", "rhythm", and "intonation" based on the specified features.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a music piece is played in real time, the output performance information (performance data such as the MIDI standard) contains a melody and other accompaniments mixed together. There is a desire to extract only the melody from these and use it when enjoying visualizing the performance or practicing the performance. However, the apparatus described in Patent Document 1 does not extract the melody from the performance data.

[0005] The present invention is for solving the problems in such a situation, and provides an information processing apparatus, a control method of the information processing apparatus, and a program that can extract only the melody from performance data. [[ID=4i]]

Means for Solving the Problems

[0006] To solve the aforementioned problems, the information processing device according to the present invention includes a control unit that receives performance information input from an input unit for inputting musical performance information as predetermined performance data, and the control unit is characterized in that it extracts a melody from the performance data based on information regarding the velocity in the performance data. [Effects of the Invention]

[0007] According to the present invention, only the melody can be extracted from performance data. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows the external appearance of the electronic musical instrument system according to the embodiment. [Figure 2] This is a block diagram showing the functional configuration of an electronic musical instrument according to an embodiment. [Figure 3] This is a block diagram showing the functional configuration of an information processing apparatus according to an embodiment. [Figure 4] This flowchart shows an example of the overall processing procedure of an information processing device. [Figure 5] Figure 4 is a flowchart illustrating an example of the melody extraction process. [Figure 6] This is an example musical score to illustrate the melody extraction process of the embodiment. [Figure 7] This is an example musical score to illustrate the melody extraction process of the embodiment. [Figure 8] This figure shows an example of an image where only the melody is emphasized when a musical performance is visualized. [Figure 9] This is an explanatory diagram for using machine learning to extract melodies. [Modes for carrying out the invention]

[0009] [First Embodiment] A first embodiment of the information processing device 200, the control method for the information processing device 200, and the program according to the present invention will be described below with reference to the drawings. In this embodiment, the information processing device 200 is connected to the electronic musical instrument 100 via a wired or wireless line, and together with the electronic musical instrument 100 constitutes the electronic musical instrument system 1. The electronic musical instrument system 1 may be connected to external amplifiers, speakers, display devices, etc., via various wireless methods or wired methods such as USB, and may be configured to appropriately output audio and images based on data from the electronic musical instrument 100 and the information processing device 200. The embodiments described below are subject to various technically preferred limitations for carrying out the present invention, but the scope of the present invention is not limited to the following embodiments and illustrated examples.

[0010] As shown in Figure 1, in this embodiment, the electronic musical instrument 100 is assumed to be an electronic keyboard instrument such as an electronic piano, synthesizer, or electronic organ. As shown in Figure 2, the electronic musical instrument 100 includes a processor 110, a keyboard 120, a key scanner 130, an operation unit 140, a sound-producing unit 150, and a communication unit 160, etc. The processor 110 includes, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The processor 110 controls the electronic musical instrument 100 by reading programs and data stored in ROM and using RAM as a work area. Specifically, when a user (performer) presses a key on the keyboard 120 (playing operation), the processor 110 generates a musical tone of the pitch corresponding to the pressed key in the sound-producing unit 150 (note on), and outputs this musical tone from the speaker 150a. Furthermore, when the user releases a key, the processor 110 causes the sound-producing unit 150 to mute the musical tone that is currently being played and is associated with the key that has been released (note off).

[0011] Here, "performance operation" refers to an operation to produce sound (in this embodiment, a key press operation). The keyboard 120 has multiple controls (performance controls) as input units for inputting musical performance information. Specifically, the controls are multiple white and black keys, and when each key is operated, performance information such as key press and key release is input. Each key is associated with a different pitch (also called a "note"). When each key of the keyboard 120 is operated, the key scanner 130 detects the input performance information as predetermined performance data such as key press, key release, and key press speed. In this embodiment, the predetermined performance data is MIDI format data (MIDI data) having a data structure compliant with the MIDI (Musical Instrument Digital Interface) standard. The faster the key press speed, the louder and stronger the sound. The information on the loudness and strength of the sound detected by the key scanner 130 is velocity information (also called velocity value). The pitch information is also called key number, key number, MIDI key, or note number. The key scanner 130 outputs performance data, including pitch information and velocity values ​​of the detected keys, to the processor 110.

[0012] The operation unit 140 is composed of key switches such as mechanical, capacitive non-contact, or membrane types, buttons, a touch panel, etc. By operating the operation unit 140, the user can, for example, set the tone or adjust the volume. The sound generation unit 150 includes a sound source, a D / A converter, an amplifier, and a speaker 150a, etc., and generates musical tones at pitches instructed by the processor 110, or mutes musical tones at pitches instructed by the processor 110. The communication unit 160 includes wireless and wired units for communicating with external devices. In this embodiment, the communication unit 160 includes an interface compliant with the MIDI standard. Based on the control of the processor 110, the communication unit 160 transmits performance information (MIDI data of the performance) detected as predetermined performance data to the information processing device 200.

[0013] In this embodiment, the information processing device 200 may be, for example, a notebook PC (Personal Computer) or a smartphone. The information processing device 200 may also be other types of devices such as a tablet terminal, a portable game console, a feature phone, or a PDA (Personal Digital Assistant). As shown in Figure 3, the information processing device 200 comprises a processor 210, RAM 220, ROM 230, an operation unit 240, a communication unit 250, a display unit 260, and an audio output unit 270. The information processing device 200 is placed, for example, on the music stand 170 of the electronic musical instrument 100. However, the information processing device 200 is not limited to being placed on a part of the electronic musical instrument 100 as shown in Figure 1, but may also be a desktop PC or the like installed separately from the electronic musical instrument 100.

[0014] A processor 210, an example of a computer, can be, for example, a single processor or a multi-processor, and includes at least one processor. In a configuration including multiple processors, the processor 210 may be packaged as a single device, or it may consist of multiple physically separated devices within the information processing device 200. The processor 210 controls the operation of each part of the information processing device 200 by reading programs and data stored in the ROM 230 into the RAM 220 and using the RAM 220 as a work area. The RAM 220 temporarily holds data and programs. The RAM 220 holds programs and data read from the ROM 230, as well as other data necessary for communication. The ROM 230 is a non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), or EEPROM (Electrically Erasable Programmable ROM), and serves as a secondary or auxiliary storage device. The ROM 230 stores system programs, various application programs, and other programs 232 and data used by the processor 210 to perform various processes.

[0015] In an embodiment, the processor 210 of the information processing apparatus 200 functions as a control unit that receives performance information of music input by keys, which are the input unit of the electronic musical instrument 100, as predetermined performance data (in the embodiment, MIDI data). In the embodiment, the program stored in the ROM 230 includes a program for extracting a melody from the performance data. The processor 210 extracts the melody in the performance data based on information (velocity value) regarding the velocity in the performance data in cooperation with such a program.

[0016] The operation unit 240 is composed of, for example, key switches, buttons, keyboards, mice, touch panels, etc. of a mechanical method, a capacitive non-contact method, a membrane method, etc. When the user operates the operation unit 240, a signal indicating the operation content is output to the processor 210. The processor 210 controls the information processing apparatus 200 based on the input signal. When the operation unit 240 includes a touch panel, the operation unit 240 also serves as the display unit 260. The communication unit 250 includes wireless and wired units for communicating with external devices. In this embodiment, the communication unit 250 includes an interface compliant with the MIDI standard. The communication unit 250 receives the input of performance information received from the electronic musical instrument 100 and stores it in the RAM 220. Further, the display unit 260 includes a display panel such as an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, an LED (Light Emitting Diode) panel, etc. and a display controller. The display unit 260 displays an image based on the output from the processor 210. The audio output unit 270 includes a speaker and a driver. The audio output unit 270 outputs audio based on the output from the processor 210.

[0017] In the embodiment, the information processing apparatus 200 includes the display unit 260 and the audio output unit 270, but the configuration of the present invention is not limited to this. At least one of the display unit 260 and the audio output unit 270 may not be provided in the information processing apparatus 200 and may be an external device for the information processing apparatus 200. When the display unit 260 is an external device, for example, a large-screen monitor (display) may be provided.

[0018] Next, the operations of the information processing apparatus, the control method of the information processing apparatus, and the program according to the embodiment will be described. FIG. 4 shows the overall flow of the processing by the information processing apparatus. The melody extraction processing in step S2 of FIG. 4 will be described in further detail using FIG. 5 and the like.

[0019] In the embodiment, each time a key-pressing operation is performed on the keyboard 120, the electronic musical instrument 100 outputs performance data (for example, note-on information of MIDI data) corresponding to the key-pressing operation to the information processing apparatus 200 via the communication unit 160. The performance data corresponding to the key-pressing operation includes at least pitch information (such as note numbers) and velocity values. Also, each time a key-releasing operation is performed on the keyboard 120, the electronic musical instrument 100 outputs performance data (for example, note-off information of MIDI data) corresponding to the key-releasing operation to the information processing apparatus 200 via the communication unit 160. The performance data corresponding to the key-releasing operation includes at least pitch information.

[0020] Specifically, first, as shown in FIG. 4, the processor 210 (control unit) of the information processing apparatus 200 determines whether there is an input (transmission) of performance data from the electronic musical instrument 100 (step S1). If there is no input of performance data (step S1: NO), the processor 210 performs a determination process as to whether the performance has ended (step S4). That is, the processor 210 determines whether a state without input of performance data has elapsed for a predetermined time. If there is no performance data for the predetermined time (step S4; YES), the processor 210 determines that the performance has ended. On the other hand, until the predetermined time elapses (step S4: NO), the processor 210 returns to step S1 and repeats the determination process.

[0021] When performance data is input from the electronic instrument 100 (Step S1: YES), the processor 210 performs processing based on the performance data (Step S2). In this embodiment, the processing based on the performance data includes, for example, determining the key, chord type, note name, etc. Specifically, the processor 210 performs a key determination, for example, determining one of 24 types from C major to B minor, and a chord determination, for example, determining Major, minor, sus4, aug, dim, 7th, etc. The processor 210 also determines the note name, for example, Do, Re, Mi, etc. The note name can be determined by the note number included in the performance data. In this embodiment, as part of the processing based on the performance data, the processor 210 extracts the melody from the performance data based on velocity information (velocity value) in the performance data.

[0022] Here, with reference to Figure 5 and other figures, the melody extraction process using velocity values ​​in the embodiment will be explained. As shown in Figure 5, when extracting a melody from performance data (MIDI data in the embodiment), the processor 210 first determines whether the input performance data contains a single note or not (step S11). A single note is, for example, when a performer presses a key with only one finger of one hand, such as the right hand. What is determined to be a melody depends on whether multiple notes are input simultaneously. In the case of a single note (step S11; YES), the processor 210 determines that the note is a melody (melody line) (step S12). On the other hand, if it is not a single note (step S11; NO), the processor 210 further determines whether there are two notes or not (step S13). If there are more than two notes (step S13; NO), the processor 210 further determines whether there are three or more notes and whether they are judged to be a chord (step S14). If there are three or more notes but they are not judged to be a chord (step S14; NO), the processor 210 proceeds to the same judgment process as in the case of two notes (step S15). Whether or not it is judged to be a chord is determined by, for example, the variation in the timing of pressing the keys of the multiple incoming notes. Even if the performer plays multiple notes simultaneously with the intention of creating a chord, there is actually a slight time difference. Therefore, if the variation (time difference) is within a predetermined range, it is judged to have been played with the intention of creating a chord, and if the variation is outside the predetermined range, it is judged not to be a chord. The predetermined range can be set as appropriate, but for example, if the variation is within about 20 milliseconds (ms), it is judged to be a chord.

[0023] If there are two notes (step S12; YES) or three or more notes but they are not judged to be a chord (step S14; NO), the processor 210 determines whether the note that was immediately judged to be a melody is sustained (step S15). For example, if the right hand presses the key for "C" and this is judged to be a melody, and the left hand presses the key for "D" while maintaining the "C" key press, i.e., without releasing the key (note off, key off), then the sounds being played at that moment will be the two notes "C" and "D". Note that when determining whether a sound is sustained, the processor does not consider whether the pedal has been pressed or not, but only the state of the key being pressed.

[0024] If the note previously identified as a melody is sustained (Step S15; YES), the processor 210 compares the velocity values ​​of the simultaneously playing notes (in the above example, "Do" and "Re") and determines whether the velocity value of the previous melody is greater than that of the current note (the newly entered note) (Step S16). Velocity refers to the initial speed at which a key is pressed, and when comparing the previous melody with the current note, the determination is made based on the velocity value, which is the initial speed at which the key is pressed. Therefore, the processor 210 identifies the note with the largest velocity value at the initial note-on point (key-press point) among the performance data as the melody, and if the velocity value of the previous melody is greater than that of the current note (Step S16; YES), it determines that the melody line remains unchanged and is the same as the previous melody (Step S17).

[0025] For example, at the beginning of the musical score shown in Figure 6, the note "A" played with a velocity of 70 (indicated by the dashed line in the figure) and three other notes played with velocities of 61 and 56 are sounding simultaneously. Even when notes are written on the score to be played simultaneously, when played by a person, there is a slight time lag between each note in the MIDI data. For example, in the example shown in Figure 6, if the note "A" is played first with a velocity of 70 and judged to be the melody, and then notes with a slight time delay are played with velocities of 61 and 56, and the "A" played with a velocity of 70 is sustained, the processor 210 will maintain the melody line at "A" and will not change the melody line to a note with a lower velocity. On the other hand, when a second sound different from the first sound, which has been determined to be a melody, is introduced while the first sound is still playing, the processor 210 compares the velocities of the first and second sounds. If the velocity of the second sound is greater than that of the first sound, the processor determines the second sound to be the melody. In other words, if the velocity of the current sound is greater than the velocity of the previous melody (step S16; NO), the processor 210 determines that the melody line has changed from the previous melody to the sound with the maximum velocity (step S18). It is conceivable to set some kind of threshold for the velocity value when determining whether or not the melody line has changed. However, in order to appropriately reflect subtle differences in key touch in determining whether or not it is a melody, it is preferable not to set a threshold and to determine the sound with the maximum velocity value as the melody if there is even a slight difference.

[0026] On the other hand, if the sound previously judged to be a melody is not sustained (step S15; NO), that is, if two or more new sounds are added, the processor 210 determines whether the velocity values ​​of the newly added sounds are the same (step S19). If the velocity values ​​of the multiple sounds are not the same (step S19; NO), the processor 210 determines the sound with the highest velocity value to be the melody (step S20). On the other hand, if the velocity values ​​of the multiple sounds are the same, that is, if the first sound and the second sound are present at the same time, the processor 210 compares the velocity values ​​of the first sound and the second sound and determines that the velocity value of the first sound and the velocity value of the second sound are the same (step S19; YES), the processor 210 determines the sound with the higher pitch among the multiple sounds to be the melody (step S21). When playing a piece of music, for example, in the musical score shown in Figure 6, the upper staff represents the part played by the right hand, and the lower staff represents the part played by the left hand. In most songs, the right hand plays the melody, while the left hand plays the bassline or other accompaniment. Therefore, generally, higher pitched notes are more likely to be the melody. For this reason, if the melody cannot be determined from the velocity value, the processor 210 will determine that the higher pitched note is the melody.

[0027] On the other hand, if there are three or more sounds playing simultaneously and they are judged to be a chord (step S14; YES), the processor 210 further determines whether the difference in velocity values ​​of each note judged to be a chord is within a predetermined value (step S22). For example, when three notes are played, if the performer plays them as a chord, the variation in velocity values ​​is small. For this reason, if the velocity values ​​of the three notes are within a predetermined value, such as 50, 52, and 55 (step S22; YES), the processor 210 determines that a chord has been played as accompaniment. Note that the predetermined value can be set as appropriate and is not limited to the 10 shown as an example. If it is determined that a chord has been played, the processor 210 further determines whether the note that was judged to be a melody immediately before is sustained (step S23). If the note that was judged to be a melody immediately before is sustained (step S23; YES), the processor 210 considers the sustained note to be a melody and maintains the melody line (step S24).

[0028] On the other hand, if the note previously identified as a melody is not sustained (Step S23; NO), the processor 210 determines that there is no melody line, assuming that only chords played as accompaniment are present (Step S25). In other words, if there are multiple notes in the performance data that are entered within a predetermined time difference, and the velocity difference at the initial note-on point of each note is below a certain value, the processor 210 determines that only chords played as accompaniment are present and the melody line is not being played, and does not identify any of the notes as a melody. Conversely, if the velocity difference of each note identified as a chord is not within a predetermined value (Step S22; NO), the processor 210 identifies the note with the highest velocity as the melody (Step S26). Thus, the processor 210 can appropriately separate and extract only the melody from other notes such as accompaniment from the performance data input in real time.

[0029] Returning to Figure 4, the processor 210 periodically determines whether a predetermined time has elapsed without input of performance data (step S4). If the predetermined time has not yet elapsed (step S4; NO), it returns to step S1 and repeats the process. On the other hand, if a predetermined time has elapsed without input of performance data (step S4; YES), the processor 210 stores the processing results of the performance data, such as the melody extraction result based on velocity values, and the determination results of key signature, chord type, note name, etc., in the storage unit 230, etc. (step S5). The processor 210 also controls each unit to perform various output processing based on the processing results (step S6). An example of output processing in this embodiment is image processing, which creates an illustration or other image from the performance data and displays it on the display unit 260. For the method of creating images from musical performance, for example, techniques disclosed in Japanese Patent Publication No. 2019-101168 (Patent No. 7035486) and Japanese Patent Publication No. 2022-148366 (Patent No. 7327434) can be used. By visually displaying the performance data as images in real time to performers, it is expected that the performers' motivation to perform and practice will be increased. In this case, if the melody can be extracted as in the embodiment, when displaying the image, the illustration corresponding to the melody line can be distinguished from illustrations derived from other performance data by, for example, surrounding it with a thick border or adding special effects such as making it sparkle like glitter. For example, Figure 8 shows an example of an image in which, when images of stars of various sizes corresponding to sounds are displayed on the display unit 260, etc., the star corresponding to the sound determined to be a melody is highlighted with a thick border. In this way, by making the image corresponding to the sound determined to be a melody different from the others, the melody line does not get buried in the image and becomes easier to follow and understand visually.

[0030] Furthermore, since the extracted melody is stored in the memory unit 230, etc., it is possible to play only the melody line later and check whether the player is correctly emphasizing the melody during their performance. For example, as shown in Figure 7, some songs have a melody line played with the left hand and an accompaniment played with the right hand. In this case, the pitch of the melody line will be lower than the pitch of the accompaniment. Even in this case, the performer will try to make the melody stand out, so if the melody is correctly emphasized, the velocity value will be higher for the melody line regardless of the pitch. However, if the velocity values ​​of multiple notes are the same and the melody cannot be determined by the velocity value, the processor 210 will generally determine that the higher-pitched note is the melody (see step S21 in Figure 5). For this reason, in the case of a song like the one shown in Figure 7, the higher-pitched note that was actually played as accompaniment will be judged as the melody. Also, if the melody is played softly, the low velocity value will prevent the melody from being correctly extracted. In this way, if the melody line is incorrectly identified, it can be explained as a misrecognition due to the inability to emphasize the melody during performance. This provides the performer with material to reflect on when they later review the melody extraction results. Therefore, it can be helpful for the performer during practice.

[0031] As described above, the information processing device 200 of this embodiment includes a processor 210, which is a control unit that receives performance information input by key pressing operations, which is an input unit for inputting musical performance information, as predetermined performance data. In this embodiment, the performance data is MIDI data conforming to the MIDI standard and includes various information such as pitch and velocity value. In this embodiment, the processor 210 extracts the melody from the performance data based on the velocity value (velocity information) in the performance data. That is, the performer plays the melody line they want to emphasize the most in the song more strongly, so the velocity value is high in the melody part. Therefore, by looking at the magnitude of the velocity value, it is possible to appropriately extract the melody even when multiple sounds are input at the same time. This makes it possible to extract only the melody from real-time performance data and use the extraction result for various processing. That is, in technology that visualizes and displays a performance, it is possible to highlight the melody line so that it is not buried in other sounds such as accompaniment, or the performer can later listen to only the melody part of their own performance to check whether they were able to correctly emphasize the melody and use it to help with lessons.

[0032] Furthermore, the processor 210 in this embodiment determines the melody from the performance data, specifically the note that shows the highest velocity value at the initial note-on point. In other words, when the note-on state continues, the velocity value is determined solely by the initial speed of the key press. For example, in this embodiment, when a second note, different from the first note which has been determined to be the melody, is played while the first note is still playing, the processor 210 compares the velocity values ​​of the first and second notes. In this comparison, the initial speed of the key press is used for both the first and second notes. If the velocity value of the second note is greater than that of the first note, the processor determines the second note to be the melody. This allows the processor to correctly extract the melody by determining the note that the performer played more strongly as the melody, and shifting the melody line if a note is played more strongly.

[0033] Furthermore, in the embodiment, when the first and second sounds are present simultaneously, the processor 210 compares the velocity values ​​of the first and second sounds. If the velocity values ​​of the first and second sounds are the same, it determines that the sound with the higher pitch is the melody. When the first and second sounds are newly introduced and their velocity values ​​are the same, a melody determination based on velocity values ​​is not possible. In this case, the melody is determined according to the general principle that the sound with the higher pitch is more likely to be the melody. This avoids situations where the melody cannot be determined. Also, if the determination of whether or not a sound is a melody is incorrect, the performer who checks the performance later can realize that their performance did not correctly emphasize the melody, providing material for reflection during subsequent practice. This allows for more effective lessons.

[0034] In another embodiment, the processor 210 is configured not to identify any of the notes as a melody if there are multiple notes in the performance data that enter within a predetermined time difference, and the difference in velocity values ​​at the initial note-on point of each note is below a certain value. When the time difference and the difference in velocity values ​​are below a predetermined value, it can be determined that the performer played chords as accompaniment. If any of the notes were identified as the melody in this case, it would deviate from the original melody line. Therefore, in such cases, by determining that only the accompaniment is present and the melody is in a rest state, it is possible to prevent picking up an incorrect melody line.

[0035] [Second Embodiment] The above describes a case where the processor 210 determines melody extraction from performance data (MIDI data) according to predetermined rules. However, the method for extracting melodies from performance data is not limited to this. To isolate and extract only the melody from performance data that contains a mixture of melody and other accompaniments, a machine learning method can be used. This involves creating a model to distinguish between melody and accompaniment, and then applying this model to inference. A method for extracting melodies from performance data using machine learning will be explained with reference to Figure 9.

[0036] First, a large amount of performance data (MIDI data) is prepared in advance, and from this data, information such as the note number (pitch), velocity value (sound intensity), and duration (sound length) of each note (note n) that makes up the performance is extracted. This is considered one-dimensional information for each note. Furthermore, for a predetermined number of notes (for example, 9 notes from note-1n to -9n) that precede a given note (note n), the note number, velocity value, and duration are similarly extracted and stored. In addition, for the notes (for example, 9 notes) preceding the predetermined number (note-1n to -9n), information about the time difference from the given note (note n) is also stored. Note that even in cases like chords, where notes are marked as overlapping on the musical score and multiple sounds are heard almost simultaneously by the human ear, there is a slight time lag between each note in the MIDI data. For this reason, even the notes that make up a chord are treated as individual notes with their own time differences.

[0037] Next, a human inputs the correct data indicating whether each note (notes -1n to -9n) is a melody or an accompaniment. As a result, if data from the 9th note prior to note n is used as described above, a 48-dimensional vector is created consisting of 5 pieces of information for each note from note -1n to -9n (i.e., note number, velocity value, duration, time difference from note n, and correct data) and 3 pieces of information for note n (i.e., note number, velocity value, and duration) ("Feature Extraction" on the left side of Figure 9). This 48-dimensional vector is generated for each note that makes up the performance data, and each is trained using machine learning ("Machine Learning" on the left side of Figure 9). Finally, a trained model is output as a result of the training ("Model Output" on the left side of Figure 9). Any type of machine learning can be used; for example, "supervised learning" such as a support vector machine (SVM) that performs classification tasks using training data is acceptable, or neural networks, deep neural networks (deep learning), etc., can be used.

[0038] Next, in the inference stage (the "Inference" section on the right in Figure 9), each item (note number, velocity value, duration, and time difference from note n) is extracted for each note obtained from the performance data (the "Feature Extraction" section on the right in Figure 9). However, since there is no ground truth data in the inference stage, the inference result is used to determine whether or not it is a melody. In this way, by inputting the extracted features into the model obtained through learning and performing inference, an inference result of whether or not each note is a melody can be obtained. Specifically, as an inference result, a value indicating melody-likeness is obtained for each note. Then, based on this inference result, a judgment result of whether or not each note is a melody is derived. For example, a threshold value indicating melody-likeness is set, and if the value obtained in inference is above the threshold, the note is judged to be a melody, and if it is below the threshold, a judgment result of whether or not it is an accompaniment or other note is output.

[0039] In the above example, we showed a set consisting of notes (notes -1n to -9n in the example above) that precede a given note (note n) by a predetermined number of notes in the time axis. However, it is not mandatory to define a set as note n + notes -1n to -9n. A note alone cannot be judged as being part of a melody; its relationship to the notes before and after it must be considered. In the above example, since a set consists of notes that precede a given note (note n) by a predetermined number of notes in the time axis, it is not possible to determine whether a note is part of a melody until the predetermined number of notes have been collected. Therefore, if the number of notes in a set is too large, the timing for initiating the determination of whether it is a melody will be delayed. On the other hand, if the number of notes in a set is too small, the accuracy of the determination may decrease. The number of notes in a set can be changed as appropriate, as long as the timing of the determination is not delayed too much and the accuracy of the determination can be ensured. Also, depending on the song, the ease of determining whether it is a melody may differ due to its complexity or fast tempo. Therefore, it is possible to prepare multiple types of learning models, such as a learning model for relatively easy-to-judge songs (e.g., songs with few notes or a slow tempo), a learning model for complex songs with a fast tempo that are difficult to judge, a learning model for classical music, and a learning model for rock and pop music, and switch between the appropriate models depending on the song to be judged to determine whether or not it is a melody. As described above, by using machine learning to extract melodies, it is possible to extract melodies from performance data with a high probability using velocity values, etc., as long as a learning model is prepared.

[0040] Although embodiments of the present invention have been described above, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. For example, in the above embodiment, we assumed that the input unit for inputting musical performance information is a key such as that of an electronic piano, and that the electronic instrument 100 is an electronic keyboard instrument. However, the electronic instrument can be any instrument that corresponds to predetermined performance data such as MIDI data, and is not limited to electronic keyboard instruments. For example, the electronic instrument could be an electronic wind instrument such as an electronic saxophone, or an electronic string instrument such as an electronic guitar.

[0041] Furthermore, the performances from which melodies are extracted from performance data using the method described in this embodiment are not limited to those performed by a single instrument. For example, melodies may be extracted when multiple electronic pianos are playing together, or when various types of electronic instruments are playing together. In other words, when multiple types of instruments are playing together, for example, when the melody line of a song that was initially played by an electronic piano is transferred to an electronic saxophone midway through, it may still be possible to follow this as a melody. Also, when multiple types of instruments are playing together in this way, when the type of instrument responsible for the melody line changes, the method of emphasizing the melody line or the color may be changed to make it visually clear which instrument is currently responsible for the melody line.

[0042] Although several embodiments of the present invention have been described above, the scope of the present invention is not limited to the embodiments described above. The specific configurations, structures, and positional relationships shown in the above embodiments can be modified as appropriate without departing from the spirit of the present invention. Furthermore, the present invention includes the scope of the invention as described in the claims and its equivalents. [Explanation of Symbols]

[0043] 210... Processor, 200... Information Processing Unit

Claims

1. The system includes a control unit that receives the performance information input from an input unit for inputting musical performance information as predetermined performance data, The control unit extracts the melody from the performance data based on information regarding the velocity in the performance data. An information processing device characterized by the following:

2. The control unit determines that the note with the highest velocity value at the initial note-on point among the performance data is the melody. The information processing apparatus according to feature 1.

3. When a second sound different from the first sound, which has been determined to be a melody, is introduced while the first sound is continuing, the control unit compares the velocity values ​​of the first sound and the second sound, and if the velocity value of the second sound is greater than that of the first sound, it determines the second sound to be the melody. The information processing apparatus according to feature 1.

4. The control unit, when both the first and second sounds are present simultaneously, compares the velocity values ​​of the first and second sounds. If the velocity values ​​of the first and second sounds are the same, it determines that the sound with the higher pitch is the melody. The information processing apparatus according to feature 1.

5. The control unit shall not determine any of the sounds as a melody if, among the performance data, there are multiple sounds that enter within a predetermined time difference, and the difference in velocity values ​​at the initial note-on point of each sound is below a certain value. The information processing apparatus according to feature 1.

6. A control method for an information processing device that receives performance information input from an input unit for inputting musical performance information as predetermined performance data, Based on the velocity information in the performance data, the melody is extracted from the performance data. A control method for an information processing device characterized by the following features.

7. The computer of the information processing device receives the performance information input from the input unit for inputting musical performance information as predetermined performance data, Based on velocity information in the performance data, it functions as a melody extraction means for extracting a melody from the performance data. A program characterized by the following features.

Images